I have some ideas to offer on this problem. Some of this will be rigorous, and some will be more intuitive but provide a basis for further investigation. So consider this a detailed exploration.
Defining Terms
Referring to the originally stated problem, we'll refer to the set $A$ that meets the given conditions as the
\textit{solution set} or sometimes simply as $A$. We'll refer to the the 3-element subset $S$ of $A$ as
\textit{the subset} or \textit{a subset} or simply $S$. And we'll call the conditions of $|n-m|
\geq \sqrt{n}+\sqrt{m}$ being applied to a pair of elements in $S$ as simply, \textit{the conditions}.
An Aside: The Initially Determined Upper Bound
Since $A \subset \mathbb{N}$ then for any two distinct positive integer elements $m, n \in A$ it's obvious
that $\sqrt{m}+\sqrt{n} > 2$. Therefore, any $S \subset A$ that satisfies the conditions must have two
elements with a difference of at least 3. It then follows that there cannot be 3 consecutive positive integers
anywhere in $A$. That leads to $|A| \leq \frac{2}{3}\times2018 < 1345$, \textit{i.e.}, $|A|
\leq 1344$.
Bounding the Problem
To determine the maximum number of elements that the solution set can have, it's helpful to characterize what
the contents of this set, $A$, look like. We'll assume that the elements of $A = \{ a_1, a_2, a_3, ..., a_N
\}$ are in increasing order so that $a_i < a_j$ if $i < j$.
The conditions of the problem dictate that we only considering 3-element subsets $S$ of $A$. In considering
such subsets, I'll argue that we only need to consider those subsets whose elements are consecutive in $A$: $S
= \{ a_i, a_{i+1}, a_{i+2} \}$ for $1 \leq i \leq N-2$. The reason is because of the following assertion that
is fairly easy to prove:
If we have $n, m \in \mathbb{N}$ and $\vert n - m\vert \geq \sqrt{n} + \sqrt{m}$, then it is true that $|n_1 - m_1| \geq \sqrt{n_1} + \sqrt{m_1}$ for $n_1 \geq n$ and $1 \leq m_1 \leq m$.
(To prove this, represent $n_1 = n + a$ and $m_1 = m - b$ for some $a, b \in \mathbb{N}_0$.)
If we can assure, then, that all subsets $S$ of $A$ that consist of 3 consecutive elements in $A$ meet the
conditions, then all 3-element subsets of $A$ will meet the conditions.
Characterizing $A$
The initially determined upper bound for $|A|$ assumed the smallest possible difference between
values in $A$ across the entire range of the values in $A$. However, due to the conditions (again, just
considering subsets of consecutive elements as we're now assuming), the difference between elements must, on
average, continue to increase as the elements themselves get larger. I say "on average" because we're
considering 3-element subsets of $A$, and the conditions are only required for one pair of those 3 elements,
which potentially allows two of the elements to have an arbitrarily small difference $\geq 1$.
If we have $A = \{a_1, a_2, a_3, ...\}$ and if we want to consider a subsets $S = \{a_i, a_{i+1}, a_{i+2}\}$,
we should apply the conditions to $a_i$ and $a_{i+2}$ in each case which will make $S$ as compact as possible
and, thus, $A$ as compact as possible and maximize the number of elements in $A$. That is, applying the
conditions to a pair of consecutive elements in $S$ creates more overall distance between the elements of $S$.
Further, when we apply the conditions, we want to be as close to equality in the condition as possible. That
is, we want, for each such $S$, that $a_{i+2}$ be the smallest value greater than $a_i$ such that $a_i$ and
$a_{i+2}$ meet the condition. The same thought process applies if we consider the subset $S' = \{a_{i+1},
a_{i+2}, a_{i+3}\}$ and so on.
Constructing $A$
Based upon the characterization above, $A$ can be constructed as two interleaving sequences, the odd and even elements,
each whose consecutive elements meet the given condition. If we want to make $A$ as compact as possible, maximizing the
number of elements, we would want to start with 1 and 2 as the initial elements, $A = \{1, 2, ...\}$.
To determine which elements should come next, we can manipulate the original condition as follows:
$$|a_{k+2} - a_k| \geq \sqrt{a_{k+2}} + \sqrt{a_k}$$
Since $a_{k+2} > a_k$:
$$a_{k+2} - a_k \geq \sqrt{a_{k+2}} + \sqrt{a_k}$$
Dividing both sized by $\sqrt{a_{k+2}} + \sqrt{a_k}$:
$$\sqrt{a_{k+2}} - \sqrt{a_k} \geq 1$$
Or:
$$a_{k+2} \geq (\sqrt{a_k}+1)^2$$
For the most compact set $A$:
$$a_{k+2} = \lceil (\sqrt{a_k}+1)^2\rceil$$
Using this formula, we get the following for $A$ when starting with $1, 2$:
$$1, 2, 4, 6, 9, 12, 16, 20, ...$$
The odd and even subsequences are geometric. In this specific case, they can be computed with the following
closed form:
$$a_k = \begin{cases}
(\frac{k+1}{2})^2, & \text{if $k$ is odd} \\[2ex]
\frac{k(k+2)}{4}, & \text{if $k$ is even}
\end{cases}$$
If we construct $A$ this way up to its maximum potential element value of 2018, we'll get:
$$1, 2, 4, 6, 9, 12, 16, 20, 25, 30, 36, 42, ..., 1849, 1892, 1936, 1980$$
The next value above $1980$ would be $2025$ which is $> 2018$.
How many elements are in this $A$? We can sum up the number of odd indexed elements and the number of even
indexed elements. These are computed by solving $n^2 = 2018$ and $n(n+1) = 2018$ for $n$:
$$|A| = \lfloor\sqrt{2018}\rfloor + \lfloor\frac{-1+\sqrt{-1+4(2018)}}{2}\rfloor = 44 + 44 = 88$$
Did We Find the Largest Set $A$?
Based upon how we constructed $A$, no additional values can be inserted and still have it meet the original
conditions. I believe the following can be shown to be true (as in, I have worked through it, but I need to go
back and check my work to be sure):
If $n, k \in \mathbb{N}$ and $n > k^2$, then the value $n = (k+1)^2$ is the smallest value of $n$ that meets the condition, $|n-k^2| \geq \sqrt{n}+k$. It can also be similarly shown that $n = (k+1)(k+2)$ is the smallest value that meets the condition, $|n-k(k+1)| \geq \sqrt{n}+\sqrt{k(k+1)}$.
In other words, our construction, assuming we start at $1, 2$, chose the subsequent values as small as
possible at each step of the process, thus maximizing the size of $A$ in that case.
You can also consider other starting values. What about $2, 3$ for example? I tried this and, using the above formula, came up with:
$$2, 3, 6, 8, 12, 15, 20, 24, 30, 35, 42, ...$$
Which is produced by the following closed form:
$$a_k =
\begin{cases}
\frac{(k+1)(k+3)}{4}, & \text{if $k$ is odd} \\[2ex]
\frac{k(k+4)}{4}, & \text{if $k$ is even}
\end{cases}$$
This leads to a set $A$ with a maximum of 87 elements.
Starting with $3, 4$ results in:
$$3, 4, 8, 9, 15, 16, 24, 25, 35, 36, ...$$
And the following closed form:
$$a_k =
\begin{cases}
\frac{(k+1)(k+5)}{4}, & \text{if $k$ is odd} \\[2ex]
(\frac{k+2}{2})^2, & \text{if $k$ is even}
\end{cases}$$
And then $A$ has a maximum of 86 elements.
As you'd expect, if you start with higher numbers as the initial elements in $A$ the maximum size of $A$ decreases further.
Conclusion
The above analysis wasn't done in full rigor, but the maximum size of $A$ that meets the conditions would appear to be 88 based upon that analysis.
