1. 29 Mar, 2020 1 commit
    • Daniel Stone's avatar
      LOCAL: Make 'All Users' space extremely magic · 281f22c2
      Daniel Stone authored
      
      
      Phriction's view policy is ancestral: in order to access /w/foo/bar/baz,
      you must be able to access /w/foo and /w/bar in addition to
      /w/foo/bar/baz itself.
      
      This is fine and makes life easy: by setting restrictive policies on
      top-level pages, we can lessen the risk of someone exposing information
      they shouldn't, by accidentally making
      /w/cold-fusion/secret-research/funding-meeting/2018-09-14 public, when
      the rest of the hierarchy is super locked down.
      
      Phriction also recently gained Spaces support, which is nice: rather
      than trying to lock down with groups and harmonise permissions, we can
      just move top-level wiki pages to a particular Space, and then we don't
      need to worry about groups.
      
      Our clients don't know Spaces even exist, which is great since it avoids
      us having to explain the two-tier permission model to them. The reason
      they don't know it exists is because if you can only see a single Space,
      then Phabricator hides the entire Spaces UI away from you. Great!
      
      Unfortunately one detail ruins everything: /w/ is a top-level page
      itself, it counts for permission checks, and it _must be in a Space_.
      So, there is no way to have wiki documents in mutually-invisible Spaces
      unless you also have a common Space, at which point the whole Spaces UI
      suddenly becomes very visible everywhere.
      
      In order to try to keep our wiki partitioned, but to not confuse our
      clients (and give them the chance to potentially expose confidential
      information!), we:
        - have a magic 'Visible to Everyone' space
        - actually hide that space from everyone with policies
        - hack policy filters to make this space visible to everyone _only
          for the purpose of checking policies on wiki objects_
        - only allow admins to change view/edit policies on the root wiki
          page (see comment for reason why)
      
      This actual patch can obviously never go anywhere near upstream, but on
      the other hand we should probably make them aware of the problem and see
      if they're interested in discussing a solution, which is probably just
      to bless the root page with magic semantics.
      Signed-off-by: Daniel Stone's avatarDaniel Stone <daniels@collabora.com>
      281f22c2
  2. 18 Jul, 2019 1 commit
    • epriestley's avatar
      In Ferret, allow documents with no title to match query terms by using LEFT... · cb4add31
      epriestley authored
      In Ferret, allow documents with no title to match query terms by using LEFT JOIN on the "title" ranking field
      
      Summary:
      Fixes T13345. See D20650. Currently, `PhabricatorCursorPagedPolicyAwareQuery` does a JOIN against the "title" field so it can apply additional ranking/ordering conditions to the query.
      
      This means that documents with no title (which don't have this field) are always excluded from the result set.
      
      We'd prefer to include them, just not give them any bonus ranking/relevance boost. Use a LEFT JOIN so they get included.
      
      Test Plan:
        - Applied D20650 (diff 1), made it use raw `getTitle()` as the document title, indexed a paste with no title.
        - Searched for a term in the paste body.
        - Before change: no results.
        - After change: found result.
      
      {F6601159}
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13345
      
      Differential Revision: https://secure.phabricator.com/D20660
      cb4add31
  3. 21 May, 2019 1 commit
    • epriestley's avatar
      Build a rough transaction-level view of Feed · 64211370
      epriestley authored
      Summary:
      Ref T13294. An install is interested in a way to easily answer audit-focused questions like "what edits were made to any Herald rule in Q1 2019?".
      
      We can answer this kind of question with a more granular version of feed that focuses on being exhaustive rather than being human-readable.
      
      This starts a rough version of it and deals with the two major tricky pieces: transactions are in a lot of different tables; and paging across them is not trivial.
      
      To solve "lots of tables", we just query every table. There's a little bit of sleight-of-hand to get this working, but nothing too awful.
      
      To solve "paging is hard", we order by "<dateCreated, phid>". The "phid" part of this order doesn't have much meaning, but it lets us put every transaction in a single, stable, global order and identify a place in that ordering given only one transaction PHID.
      
      Test Plan: {F6463076}
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13294
      
      Differential Revision: https://secure.phabricator.com/D20531
      64211370
  4. 01 Apr, 2019 2 commits
    • epriestley's avatar
      (stable) Correct some straggling Ferret/Cursor interactions · 4105278d
      epriestley authored
      Summary:
      See PHI1182. Ref T13266. The recent fixes didn't quite cover the case where you have a query, but order by something other than relevance, and page forward.
      
      Refine the tests around building/selecting these columns and paging values a little bit to be more specific about what they care about.
      
      Test Plan:
      Executed queries, then went to "Next Page", for:
      
        - query text, non-relevance order.
        - query text, relevance order.
        - no query text, non-relevance order.
        - no query text, relevance order.
      
      Also, made an API call similar to the one in PHI1182.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13266
      
      Differential Revision: https://secure.phabricator.com/D20354
      4105278d
    • epriestley's avatar
      Correct some straggling Ferret/Cursor interactions · 00b1c419
      epriestley authored
      Summary:
      See PHI1182. Ref T13266. The recent fixes didn't quite cover the case where you have a query, but order by something other than relevance, and page forward.
      
      Refine the tests around building/selecting these columns and paging values a little bit to be more specific about what they care about.
      
      Test Plan:
      Executed queries, then went to "Next Page", for:
      
        - query text, non-relevance order.
        - query text, relevance order.
        - no query text, non-relevance order.
        - no query text, relevance order.
      
      Also, made an API call similar to the one in PHI1182.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13266
      
      Differential Revision: https://secure.phabricator.com/D20354
      00b1c419
  5. 25 Mar, 2019 1 commit
    • epriestley's avatar
      When paging by Ferret "rank", page using "HAVING rank > ...", not "WHERE rank > ..." · 7f905706
      epriestley authored
      Summary:
      Ref T13091. The Ferret "rank" column is a function of the query text and looks something like `SELECT ..., 2 + 2 AS rank, ...`.
      
      You can't apply conditions to this kind of dynamic column with a WHERE clause: you get a slightly unhelpful error like "column rank unknown in where clause". You must use HAVING:
      
      ```
      mysql> SELECT 2 + 2 AS x WHERE x = 4;
      ERROR 1054 (42S22): Unknown column 'x' in 'where clause'
      mysql> SELECT 2 + 2 AS x HAVING x = 4;
      +---+
      | x |
      +---+
      | 4 |
      +---+
      1 row in set (0.00 sec)
      ```
      
      Add a flag to paging column definitions to let them specify that they must be applied with HAVING, then apply the whole paging clause with HAVING if any column requires HAVING.
      
      Test Plan:
        - In Maniphest, ran a fulltext search matching more than 100 results, ordered by "Relevance", then clicked "Next Page".
        - Before patch: query with `... WHERE rank > 123 OR ...` caused MySQL error because `rank` is not a WHERE-able column.
        - After patch: query builds as `... HAVING rank > 123 OR ...`, pages properly, no MySQL error.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13091
      
      Differential Revision: https://secure.phabricator.com/D20298
      7f905706
  6. 19 Mar, 2019 6 commits
    • epriestley's avatar
      Select Ferret fulltext columns in results so fulltext queries work under UNION · b90e02be
      epriestley authored
      Summary:
      Ref T13091. In Differential, if you provide a query and "Sort by: Relevance", we build a query like this:
      
      ```
      ((SELECT revision.* FROM ... ORDER BY rank) UNION ALL (SELECT revision.* FROM ... ORDER BY rank)) ORDER BY rank
      ```
      
      The internal "ORDER BY rank" is technically redundant (probably?), but doesn't hurt anything, and makes construction easier.
      
      The problem is that the outer "ORDER BY rank" at the end, which attempts to order the results of the two parts of the UNION, can't actually order them, since `rank` wasn't selected.
      
      (The column isn't actually "rank", which //is// selected -- it's the document modified/created subcolumns, which are not.)
      
      To fix this, actually select the fulltext columns into the result set.
      
      Test Plan:
        - Ran a non-empty fulltext query in Differential with "Bucket: Required Action" selected so the UNION construction fired.
        - Ran normal queries in Maniphest and global search.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13091
      
      Differential Revision: https://secure.phabricator.com/D20297
      b90e02be
    • epriestley's avatar
      Skip Ferret fulltext columns in "ORDER BY" if there's no fulltext query · a8ba8217
      epriestley authored
      Summary:
      Ref T13091. If you "Order By: Relevance" but don't actually specify a query, we currently raise a bare exception.
      
      This operation is sort of silly/pointless, but it seems like it's probably best to just return the results for the other constraints in the fallback order (usually, by ID). Alternatively, we could raise a non-bare exception here ("You need to provide a fulltext query to order by relevance.")
      
      Test Plan: Queried tasks by relevance with no actual query text.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13091
      
      Differential Revision: https://secure.phabricator.com/D20296
      a8ba8217
    • epriestley's avatar
      Make the UI when you use an invalid cursor ("?after=19874189471232892") a little nicer · 3940c8e1
      epriestley authored
      Summary:
      Ref T13259. Currently, visiting a page that executes a query with an invalid cursor raises a bare exception that escapes to top level.
      
      Catch this a little sooner and tailor the page a bit.
      
      Test Plan: Visited `/maniphest/?after=335234234223`, saw a nicer exception page.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13259
      
      Differential Revision: https://secure.phabricator.com/D20295
      3940c8e1
    • epriestley's avatar
      Convert complex query subclasses to use internal cursors · 8449c179
      epriestley authored
      Summary:
      Depends on D20292. Ref T13259. This converts the rest of the `getPagingValueMap()` callsites to operate on internal cursors instead.
      
      These are pretty one-off for the most part, so I'll annotate them inline.
      
      Test Plan:
        - Grouped tasks by project, sorted by title, paged through them, saw consistent outcomes.
        - Queried edges with "edge.search", paged through them using the "after" cursor.
        - Poked around the other stuff without catching any brokenness.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13259
      
      Differential Revision: https://secure.phabricator.com/D20293
      8449c179
    • epriestley's avatar
      Convert simple query subclasses to use internal cursors · d4847c3e
      epriestley authored
      Summary:
      Depends on D20291. Ref T13259. Move all the simple cases (where paging depends only on the partial object and does not depend on keys) to a simple wrapper.
      
      This leaves a smaller set of more complex cases where we care about external data or which keys were requested that I'll convert in followups.
      
      Test Plan: Poked at things, but a lot of stuff is still broken until everything is converted.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
      
      Maniphest Tasks: T13259
      
      Differential Revision: https://secure.phabricator.com/D20292
      d4847c3e
    • epriestley's avatar
      Separate internal and external Query Cursors more cleanly, to fix pagination against broken objects · 1b0ef439
      epriestley authored
      Summary:
      Ref T13259.
      
      (NOTE) This is "infrastructure/guts" only and breaks some stuff in Query subclasses. I'll fix that stuff in a followup, it's just going to be a larger diff that's mostly mechanical.
      
      When a user clicks "Next Page" on a tasks view and gets `?after=100`, we want to show them the next 100 //visible// tasks. It's possible that tasks 1-100 are visible, but tasks 101-788 are not, and the next visible task is 789.
      
      We load task ID `100` first, to make sure they can actually see it: you aren't allowed to page based on objects you can't see. If we let you, you could use "order=title&after=100", plus creative retitling of tasks, to discover the title of task 100: create tasks named "A", "B", etc., and see which one is returned first "after" task 100. If it's "D", you know task 100 must start with "C".
      
      Assume the user can see task 100. We run a query like `id > 100` to get the next 100 tasks.
      
      However, it's possible that few (or none) of these tasks can be seen. If the next visible task is 789, none of the tasks in the next page of results will survive policy filtering.
      
      So, for queries after the initial query, we need to be able to page based on tasks that the user can not see: we want to be able to issue `id > 100`, then `id > 200`, and so on, until we overheat or find a page of results (if 789-889 are visible, we'll make it there before overheating).
      
      Currently, we do this in a not-so-great way:
      
        - We pass the external cursor (`100`) directly to the subquery.
        - We query for that object using `getPagingViewer()`, which is a piece of magic that returns the real viewer on the first page and the omnipotent viewer on the 2nd..nth page. This is very sketchy.
        - The subquery builds paging values based on that object (`array('id' => 100)`).
        - We turn the last result from the subquery back into an external cursor (`200`) and save it for the next time.
      
      Note that the last step happens BEFORE policy (and other) filtering.
      
      The problems with this are:
      
        - The phantom-schrodinger's-omnipotent-viewer thing isn't explicity bad, but it's sketchy and generally not good. It feels like it could easily lead to a mistake or bug eventually.
        - We issue an extra query each time we page results, to convert the external cursor back into a map (`100`, `200`, `300`, etc).
        - In T13259, there's a new problem: this only works if the object is filtered out for policy reasons and the omnipotent viewer can still see it. It doesn't work if the object is filtered for some other reason.
      
      To expand on the third point: in T13259, we hit a case where 100+ consecutive objects are broken (they point to a nonexistent `repositoryID`). These objects get filtered unconditionally. It doesn't matter if the viewer is omnipotent or not.
      
      In that case: we set the next external cursor from the raw results (e.g., `200`). Then we try to load it (using the omnipotent viewer) to turn it into a map of values for paging. This fails because the object isn't loadable, even as the omnipotent viewer.
      
      ---
      
      To fix this stuff, the new approach steps back a little bit. Primarily, I'm separating "external cursors" from "internal cursors".
      
      An "External Cursor" is a string that we can pass in `?after=X` URIs. It generally identifies an object which the user can see.
      
      An "Internal Cursor" is a raw result from `loadPage()`, i.e. before policy filtering. Usually, (but not always) this is a `LiskDAO` object that doesn't have anything attached yet and hasn't been policy filtered.
      
      We now do this, broadly:
      
        - Convert the external cursor to an internal cursor.
        - Execute the query using internal cursors.
        - If necessary, convert the last visible result back into an external cursor at the very end.
      
      This fixes all the problems:
      
        - Sketchy Omnipotent Viewer: We no longer ever use an omnipotent viewer. (We pick cursors out of the result set earlier, instead.)
        - Too Many Queries: We only issue one query at the beginning, when going from "external" to "internal". This query is generally unavoidable since we need to make sure the viewer can see the object and that it's a real / legitimate object. We no longer have to query an extra time for each page.
        - Total Failure on Invalid Objects: we now page directly with objects out of `loadPage()`, before any filtering, so we can page over invisible or invalid objects without issues.
      
      This change switches us over to internal/external cursors, and makes simple cases (ID-based ordering) work correctly. It doesn't work for complex cases yet since subclasses don't know how to get paging values out of an internal cursor yet. I'll update those in a followup.
      
      Test Plan: For now, poked around a bit. Some stuff is broken, but normal ID-based lists load correctly and page properly. See next diff for a more detailed test plan.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13259
      
      Differential Revision: https://secure.phabricator.com/D20291
      1b0ef439
  7. 01 Mar, 2019 1 commit
  8. 28 Jan, 2019 1 commit
    • epriestley's avatar
      Don't put "spacePHID IN (...)" constraints in queries which will raise policy exceptions · 03ac59a8
      epriestley authored
      Summary:
      See T13240. Ref T13242. When we're issuing a query that will raise policy exceptions (i.e., give the user a "You Shall Not Pass" dialog if they can not see objects it loads), don't do space filtering in MySQL: when objects are filtered out in MySQL, we can't distinguish between "bad/invalid ID/object" and "policy filter", so we can't raise a policy exception.
      
      This leads to cases where viewing an object shows "You Shall Not Pass" if you can't see it for any non-Spaces reason, but "404" if the reason is Spaces.
      
      There's no product reason for this, it's just that `spacePHID IN (...)` is important for non-policy-raising queries (like a list of tasks) to reduce how much application filtering we need to do.
      
      Test Plan:
      Before:
      
      ```
      $ git pull
      phabricator-ssh-exec: No repository "spellbook" exists!
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      ```
      
      After:
      
      ```
      $ git pull
      phabricator-ssh-exec: [You Shall Not Pass: Unknown Object (Repository)] This object is in a space you do not have permission to access.
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      ```
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13242
      
      Differential Revision: https://secure.phabricator.com/D20042
      03ac59a8
  9. 16 Nov, 2018 1 commit
    • epriestley's avatar
      Continue cleaning up queries in the wake of changes to "%Q" · 933462b4
      epriestley authored
      Summary: Depends on D19810. Ref T13217. Ref T13216. I mostly used `grep implode | grep OR` and `grep implode | grep AND` to find these -- not totally exhaustive but should be a big chunk of the callsites that are missing `%LO` / `%LA`.
      
      Test Plan:
      These are tricky to test exhaustively, but I made an attempt to hit most of them:
      
      - Browsed Almanac interfaces.
      - Created/browsed Calendar events.
      - Enabled/disabled/showed the lock log.
      - Browsed repositories.
      - Loaded Facts UI.
      - Poked at Multimeter.
      - Used typeahead for users and projects.
      - Browsed Phriction.
      - Ran various fulltext searches.
      
      Not sure these are reachable:
      
      - All the lint stuff might be dead/unreachable/nonfunctional?
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Subscribers: yelirekim
      
      Maniphest Tasks: T13217, T13216
      
      Differential Revision: https://secure.phabricator.com/D19814
      933462b4
  10. 15 Nov, 2018 3 commits
    • epriestley's avatar
      Fix all query warnings in "arc unit --everything" · 86fd2041
      epriestley authored
      Summary:
      Ref T13216. Ref T13217. Depends on D19800. This fixes all of the remaining query warnings that pop up when you run "arc unit --everything".
      
      There's likely still quite a bit of stuff lurking around, but hopefully this covers a big set of the most common queries.
      
      Test Plan: Ran `arc unit --everything`. Before change: lots of query warnings. After change: no query warnings.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13217, T13216
      
      Differential Revision: https://secure.phabricator.com/D19801
      86fd2041
    • epriestley's avatar
      Continue making application fixes to Phabricator for changes to %Q semantics · 2f10d4ad
      epriestley authored
      Summary: Depends on D19789. Ref T13217. Continue updating things to use the new %Q-flavored conversions instead of smushing a bunch of strings together.
      
      Test Plan: Browsed around, far fewer errors. These changes are largely mechanical in nature.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13217
      
      Differential Revision: https://secure.phabricator.com/D19790
      2f10d4ad
    • epriestley's avatar
      Update many Phabricator queries for new %Q query semantics · 98690ee3
      epriestley authored
      Summary: Depends on D19785. Ref T13217. This converts many of the most common clause construction pathways to the new %Q / %LQ / %LO / %LA / %LJ semantics.
      
      Test Plan: Browsed around a bunch, saw fewer warnings and no obvious behavioral errors. The transformations here are generally mechanical (although I did them by hand).
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Subscribers: hach-que
      
      Maniphest Tasks: T13217
      
      Differential Revision: https://secure.phabricator.com/D19789
      98690ee3
  11. 14 Nov, 2018 1 commit
  12. 23 Jul, 2018 1 commit
    • epriestley's avatar
      Support the Ferret "=" (exact match) operator in the actual query engine · 71d4fa41
      epriestley authored
      Summary:
      Ref PHI778. In D18492, I added support for parsing this operator, but did not actually implement it in the query engine.
      
      Implementation is fairly straightforward. This supports querying for objects by exact title with `title:="exact title"`. This is probably a bad idea, but sometimes maybe useful anyway.
      
      Test Plan: Queried for `title:="xxx"`, found only exact matches.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Subscribers: ahoffer2
      
      Differential Revision: https://secure.phabricator.com/D19529
      71d4fa41
  13. 22 Feb, 2018 1 commit
  14. 23 Oct, 2017 2 commits
    • epriestley's avatar
      Clean up virtual "_ft_rank" column for query construction of Ferret objects · 1d213dc1
      epriestley authored
      Summary:
      Ref T12974. Ferret object queries SELECT a virtual "_ft_rank" column for relevance ordering.
      
      Currently, they always SELECT this column. That's fine and doesn't hurt anything, but makes developing and debugging things kind of a pain since every query has this `, blah blah _ft_rank` junk.
      
      Instead, construct this column only if we're actually going to use it.
      
      Mostly, this cleans up DarkConsole / query logs a bit.
      
      Test Plan:
      Viewed normal query results on various pages, viewed global search results, ordered Maniphest tasks by normal stuff and by "Relevance".
      
      Viewed DarkConsole, saw no more "_ft_rank" junk on normal pages.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T12974
      
      Differential Revision: https://secure.phabricator.com/D18728
      1d213dc1
    • epriestley's avatar
      Rewrite CommitQuery to use UNION for performance · 157f47cd
      epriestley authored
      Summary:
      Ref T12680. See PHI167. See that task for discussion.
      
      Rewrite `DiffusionCommitQuery` to work more like `DifferentialRevisionQuery`, and use a UNION to find "all revisions you need to audit OR respond to".
      
      I tried to get this working a little more cleanly than RevisionQuery does, and can probably simplify that now.
      
      Test Plan: Poked at the UI locally without hitting any apparent issues, but my local data is pretty garbage at this point. I'll take a look at how the query plans work on `secure`.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T12680
      
      Differential Revision: https://secure.phabricator.com/D18722
      157f47cd
  15. 09 Oct, 2017 1 commit
    • Dmitri Iouchtchenko's avatar
      Fix spelling · 9bd6a370
      Dmitri Iouchtchenko authored
      Summary: Noticed a couple of typos in the docs, and then things got out of hand.
      
      Test Plan:
        - Stared at the words until my eyes watered and the letters began to swim on the screen.
        - Consulted a dictionary.
      
      Reviewers: #blessed_reviewers, epriestley
      
      Reviewed By: #blessed_reviewers, epriestley
      
      Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
      
      Differential Revision: https://secure.phabricator.com/D18693
      9bd6a370
  16. 03 Oct, 2017 1 commit
    • epriestley's avatar
      Allow the Ferret engine to remove "common" ngrams from the index · 1de130c9
      epriestley authored
      Summary:
      Ref T13000. This adds support for tracking "common" ngrams, which occur in too many documents to be useful as part of the ngram index.
      
      If an ngram is listed in the "common" table, it won't be written when indexing documents, or queried for when searching for them.
      
      In this change, nothing actually writes to the "common" table. I'll start writing to the table in a followup change.
      
      Specifically, I plan to do this:
      
        - A new GC process updates the "common" table periodically, by writing ngrams which appear in more than X% of documents to it, for some value of X, if there are at least a minimum number of documents (maybe like 4,000).
        - A new GC process deletes ngrams that have been added to the common table from the existing indexes.
      
      Hopefully, this will pare down the ngrams index to something reasonable over time without requiring any manual tuning.
      
      Test Plan:
        - Ran some queries and indexes.
        - Manually inserted ngrams `xxx` and `yyy` into the ngrams table, searched and indexed, saw them ignored as viable ngrams for search/index.
      
      Reviewers: amckinley
      
      Reviewed By: amckinley
      
      Maniphest Tasks: T13000
      
      Differential Revision: https://secure.phabricator.com/D18672
      1de130c9
  17. 18 Sep, 2017 1 commit
  18. 12 Sep, 2017 2 commits
    • epriestley's avatar
      Fix an issue with selecting the right stemmed ngrams with Ferret engine queries · fdc0d8c2
      epriestley authored
      Summary:
      Ref T12819. In D18581, I corrected one bug (ngram selection for terms) but introduced a minor new bug. We now pass `' query '` (term corpus with boundary spaces) to the stemmer, but it bails out on this since English words don't start with spaces.
      
      Trim these extra boundary spaces off before invoking the stemmer.
      
      The practical effect of this is that searching for non-stem variations of a word ("detection") now finds stemmed variations again ("detect"). Prior to fixing this bug, the stem could find longer variations but not the other way around.
      
      Test Plan: Searched for "detection", found results matching "detect" after patch (and saw same results for "detect" and "detection").
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18593
      fdc0d8c2
    • epriestley's avatar
      Return fulltext tokens from the Ferret fulltext engine · 39b74572
      epriestley authored
      Summary:
      Ref T12819. These render the little "Searched For: X, Y, U V" hint about how something was parsed.
      
      (This might get a "substring" color or "title only" color or something in the future.)
      
      Test Plan: {F5178807}
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18589
      39b74572
  19. 08 Sep, 2017 1 commit
    • epriestley's avatar
      When selecting Ferret ngrams, select term ngrams (not raw ngrams) for term search · c662dda0
      epriestley authored
      Summary:
      Ref T12819. For queries like `v0.2`, we would incorrectly search for ngrams including `0.2`, but this is only a substring ngram: the term corpus splits this into `v0` and `2`, so `0.2` is not in the ngrams table.
      
      When executing term queries, search for term ngrams instead. This makes "v0.2" work properly again.
      
      Test Plan: Searched for "v0.2", found a task with "v0.2" in the title.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18581
      c662dda0
  20. 07 Sep, 2017 3 commits
    • epriestley's avatar
      Reduce the amount of boilerplate that implementing FerretInterface requires · 2218caee
      epriestley authored
      Summary:
      See brief discussion in D18554. All the index tables are the same for every application (and, at this point, seem unlikely to change) and we never actually pass these objects around (they're only used internally).
      
      In some other cases (like Transactions) not every application has the same tables (for example, Differential has extra field for inline comments), and/or we pass the objects around (lots of stuff uses `$xactions` directly).
      
      However, in this case, and in Edges, we don't interact with any representation of the database state directly in much of the code, and it doesn't change from application to application.
      
      Just automatically define document, field, and ngram tables for anything which implements `FerretInterface`. This makes the query and index logic a tiny bit messier but lets us delete a ton of boilerplate classes.
      
      Test Plan: Indexed objects, searched for objects. Same results as before with much less code. Ran `bin/storage upgrade`, got a clean bill of health.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18559
      2218caee
    • epriestley's avatar
      Sort global fulltext results by overall relevance · a2a2b3f7
      epriestley authored
      Summary:
      Ref T12819. Currently, under the Ferret engine, we query each application's index separately and then aggregate the results.
      
      At the moment, results are aggregated by type first, then by actual rank. For example, all the revisions appear first, then all the tasks.
      
      Instead, surface the internal ranking data from the underlying query and sort by it.
      
      Test Plan: Searched for "A B" with a task named "A B" and a revision named "A". Saw task first. Broadly, saw mixed task and revision order in result sets.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18551
      a2a2b3f7
    • epriestley's avatar
      Use the Ferret engine fulltext document table to drive auxiliary fulltext constraints · 8059db89
      epriestley authored
      Summary:
      Ref T12819. I started trying to get individual engines to drive these constraints (e.g., `ManiphestTaskQuery` can do most of the work) but this is a big pain, especially since most engines don't support "any owner" or "no owner", and not everything has an owner, and so on and so on. Going down this path would have meant a huge pile of stub functions everywhere, I think.
      
      Instead, drive these through the main engine using the fulltext document table, which already has everything we need to apply these constraints in a uniform way.
      
      Also tweak some parts of query construction and result ordering.
      
      Test Plan: Searched for documents by author, owner, unowned, any owner, tags, subscribers, fulltext in global search. Got sensible results without any application-specific code.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18550
      8059db89
  21. 06 Sep, 2017 1 commit
  22. 05 Sep, 2017 4 commits
    • epriestley's avatar
      Add support for relevance-ranking Ferret engine results · 64b7778f
      epriestley authored
      Summary: Ref T12819. "Relevance" here just means "how many of your search terms are present in the title?" but that's about the best we can do anyway.
      
      Test Plan: Indexed tasks "A B", "A Z", "Z B", and "Z Z" (all with "A B" in comments). Searched for "A B". Got results ranked in the listed order, with "A B" as the most relevant hit for query "A B".
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18539
      64b7778f
    • epriestley's avatar
      Move Ferret engine "title:..." field definitions to the engine itself · 20aad35e
      epriestley authored
      Summary: Ref T12819. Move these out of the core engine into the Ferret engine. In the future different applications can define different functions, like "summary:..." or whatever. This may get more formalization when I possibly do "author:" and such some time down the road.
      
      Test Plan: Searched for "title:...". Searched for "dog:...", got a useful error.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18536
      20aad35e
    • epriestley's avatar
      Reduce the number of magic strings in the Ferret implementation · 46abc111
      epriestley authored
      Summary:
      Ref T12819. Push more of the magic `' '` stuff into the engine and simplify calls to ngram construction.
      
      Also fixes a bug where a task with title "apple banana" and description "cherry doughnut" could match query "banana cherry" by separating separate term segments with newlines instead of spaces.
      
      Test Plan:
        - Indexed some objects.
        - Searched (term, substring, quoted terms).
        - Viewed index in database.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18534
      46abc111
    • epriestley's avatar
      Consolidate more Ferret engine code into FerretEngine · 4a7593f4
      epriestley authored
      Summary: Ref T12819. Earlier I separated some ngram code into an "ngram engine" hoping to share it across the simple Ngrams stuff and the full Ferret stuff, but they actually use slightly different rules. Just pull more of this stuff into FerretEngine to reduce the number of moving pieces and the amount of code duplication.
      
      Test Plan: Searched for terms, rebuilt indexes.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18533
      4a7593f4
  23. 01 Sep, 2017 1 commit
  24. 30 Aug, 2017 2 commits
    • epriestley's avatar
      Add "title:..." support to the Ferret engine · 3b43a707
      epriestley authored
      Summary:
      Ref T12819. Adds (hacky, hard-coded) field support (for now, only for "title").
      
      I've written this so `title:quick ferret` is the same as `title:quick title:ferret`. I think this is what users probably mean.
      
      You can do the other thing as `ferret title:quick`, or `title:quick all:ferret`.
      
      Test Plan: Searched for `title:x`, `title:"x"`, `title:~"x"`, etc. Searched for "garbage:y", got an exception since that's not a recognized function. Searched for `title:x y`, saw both do title search.
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18503
      3b43a707
    • epriestley's avatar
      Support "-term" in Ferret engine queries · 048aa36c
      epriestley authored
      Summary:
      Ref T12819. Supports negating search terms, e.g. "apple -honeycrisp".
      
      When negating a term, we're a little more strict about what can match (that is, what can //prevent// a document from being returned) since it's easy for a user to type "apple -honeycrisp -honey -crisp -crispies -olcrispers -honeyyums" to keep refining their search, but hard/impossible to split apart an overboard term.
      
      Test Plan:
        - Ran `apple -smith`, `apple -"granny smith"`, etc.
        - Verified `phone -tact` does not exclude `phone contact`.
        - (In theory, `phone -~tact` would, but the parser currently doesn't support this, and I'm not champing at the bit to add support.)
      
      Reviewers: chad
      
      Reviewed By: chad
      
      Maniphest Tasks: T12819
      
      Differential Revision: https://secure.phabricator.com/D18502
      048aa36c