Commit Graph

75 Commits

Author SHA1 Message Date
Steve Howell 979dcfe85b refactor: Extract settings_data.py.
This extracts a new module with three
functions, which we will test with 100%
line coverage:

    - show_email
    - email_for_user_settings
    - get_time_preferences

The first two break several dependencies
in the codebase on `settings_org.js`.  The
`get_time_preferences` breaks an annoying
dependency on `page_params` within people.

The module is pretty cohesive, in terms that
all three functions are just light wrappers
around `page_params` and/or `settings_config`.

Now all the modules that want to call show_email()
only have to require `settings_data`, instead of
having a dependency on the much heavier
`settings_org.js` module.

I also make some of the unit tests here be more
full-stack, where instead of stubbing show_email,
I basically just toggle `page_params.is_admin`.
2020-02-28 17:11:24 -08:00
Steve Howell b994889315 node tests: Just set i18n every time.
Explicitly stubbing i18n in 48 different files
is mostly busy work at this point, and it doesn't
provide much signal, since often it's invoked
only to satisfy transitive dependencies.
2020-02-28 17:11:24 -08:00
Steve Howell 9ab07d1038 util.js: Remove util from window.
We now treat util like a leaf module and
use "require" to import it everywhere it's used.

An earlier version of this commit moved
util into our "shared" library, but we
decided to wait on that.  Once we're ready
to do that, we should only need to do a
simple search/replace on various
require/zrequire statements plus a small
tweak to one of the custom linter checks.

It turns out we don't really need util.js
for our most immediate code-sharing goal,
which is to reuse our markdown code on
mobile.  There's a little bit of cleanup
still remaining to break the dependency,
but it's minor.

The util module still calls the global
blueslip module in one place, but that
code is about to be removed in the next
few commits.

I am pretty confident that once we start
sharing things like the typeahead code
more aggressively, we'll start having
dependencies on util.  The module is barely
more than 300 lines long, so we'll probably
just move the whole thing into shared
rather than break it apart.  Also, we
can continue to nibble away at the
cruftier parts of the module.
2020-02-15 12:20:20 -08:00
Anders Kaseorg 7c6ec51e5d search_suggestions: Convert lookup_table from object to Map.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:39:01 -08:00
Anders Kaseorg 719546641f js: Convert a.indexOf(…) !== -1 to a.includes(…).
Babel polyfills this for us for Internet Explorer.

import * as babelParser from "recast/parsers/babel";
import * as recast from "recast";
import * as tsParser from "recast/parsers/typescript";
import { builders as b, namedTypes as n } from "ast-types";
import K from "ast-types/gen/kinds";
import fs from "fs";
import path from "path";
import process from "process";

const checkExpression = (node: n.Node): node is K.ExpressionKind =>
  n.Expression.check(node);

for (const file of process.argv.slice(2)) {
  console.log("Parsing", file);
  const ast = recast.parse(fs.readFileSync(file, { encoding: "utf8" }), {
    parser: path.extname(file) === ".ts" ? tsParser : babelParser,
  });
  let changed = false;

  recast.visit(ast, {
    visitBinaryExpression(path) {
      const { operator, left, right } = path.node;
      if (
        n.CallExpression.check(left) &&
        n.MemberExpression.check(left.callee) &&
        !left.callee.computed &&
        n.Identifier.check(left.callee.property) &&
        left.callee.property.name === "indexOf" &&
        left.arguments.length === 1 &&
        checkExpression(left.arguments[0]) &&
        ((["===", "!==", "==", "!=", ">", "<="].includes(operator) &&
          n.UnaryExpression.check(right) &&
          right.operator == "-" &&
          n.Literal.check(right.argument) &&
          right.argument.value === 1) ||
          ([">=", "<"].includes(operator) &&
            n.Literal.check(right) &&
            right.value === 0))
      ) {
        const test = b.callExpression(
          b.memberExpression(left.callee.object, b.identifier("includes")),
          [left.arguments[0]]
        );
        path.replace(
          ["!==", "!=", ">", ">="].includes(operator)
            ? test
            : b.unaryExpression("!", test)
        );
        changed = true;
      }
      this.traverse(path);
    },
  });

  if (changed) {
    console.log("Writing", file);
    fs.writeFileSync(file, recast.print(ast).code, { encoding: "utf8" });
  }
}

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-10 14:08:12 -08:00
Anders Kaseorg 11b5d80800 tests: Fix more undefined mocks.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-02-10 14:08:12 -08:00
Anders Kaseorg 02511bff1c js: Automatically convert _.each to for…of.
This commit was automatically generated by the following script,
followed by lint --fix and a few small manual lint-related cleanups.

import * as babelParser from "recast/parsers/babel";
import * as recast from "recast";
import * as tsParser from "recast/parsers/typescript";
import { builders as b, namedTypes as n } from "ast-types";
import { Context } from "ast-types/lib/path-visitor";
import K from "ast-types/gen/kinds";
import { NodePath } from "ast-types/lib/node-path";
import assert from "assert";
import fs from "fs";
import path from "path";
import process from "process";

const checkExpression = (node: n.Node): node is K.ExpressionKind =>
  n.Expression.check(node);
const checkStatement = (node: n.Node): node is K.StatementKind =>
  n.Statement.check(node);

for (const file of process.argv.slice(2)) {
  console.log("Parsing", file);
  const ast = recast.parse(fs.readFileSync(file, { encoding: "utf8" }), {
    parser: path.extname(file) === ".ts" ? tsParser : babelParser,
  });
  let changed = false;
  let inLoop = false;
  let replaceReturn = false;

  const visitLoop = (...args: string[]) =>
    function(this: Context, path: NodePath) {
      for (const arg of args) {
        this.visit(path.get(arg));
      }
      const old = { inLoop };
      inLoop = true;
      this.visit(path.get("body"));
      inLoop = old.inLoop;
      return false;
    };

  recast.visit(ast, {
    visitDoWhileStatement: visitLoop("test"),

    visitExpressionStatement(path) {
      const { expression, comments } = path.node;
      let valueOnly;
      if (
        n.CallExpression.check(expression) &&
        n.MemberExpression.check(expression.callee) &&
        !expression.callee.computed &&
        n.Identifier.check(expression.callee.object) &&
        expression.callee.object.name === "_" &&
        n.Identifier.check(expression.callee.property) &&
        ["each", "forEach"].includes(expression.callee.property.name) &&
        [2, 3].includes(expression.arguments.length) &&
        checkExpression(expression.arguments[0]) &&
        (n.FunctionExpression.check(expression.arguments[1]) ||
          n.ArrowFunctionExpression.check(expression.arguments[1])) &&
        [1, 2].includes(expression.arguments[1].params.length) &&
        n.Identifier.check(expression.arguments[1].params[0]) &&
        ((valueOnly = expression.arguments[1].params[1] === undefined) ||
          n.Identifier.check(expression.arguments[1].params[1])) &&
        (expression.arguments[2] === undefined ||
          n.ThisExpression.check(expression.arguments[2]))
      ) {
        const old = { inLoop, replaceReturn };
        inLoop = false;
        replaceReturn = true;
        this.visit(
          path
            .get("expression")
            .get("arguments")
            .get(1)
            .get("body")
        );
        inLoop = old.inLoop;
        replaceReturn = old.replaceReturn;

        const [right, { body, params }] = expression.arguments;
        const loop = b.forOfStatement(
          b.variableDeclaration("let", [
            b.variableDeclarator(
              valueOnly ? params[0] : b.arrayPattern([params[1], params[0]])
            ),
          ]),
          valueOnly
            ? right
            : b.callExpression(
                b.memberExpression(right, b.identifier("entries")),
                []
              ),
          checkStatement(body) ? body : b.expressionStatement(body)
        );
        loop.comments = comments;
        path.replace(loop);
        changed = true;
      }
      this.traverse(path);
    },

    visitForStatement: visitLoop("init", "test", "update"),

    visitForInStatement: visitLoop("left", "right"),

    visitForOfStatement: visitLoop("left", "right"),

    visitFunction(path) {
      this.visit(path.get("params"));
      const old = { replaceReturn };
      replaceReturn = false;
      this.visit(path.get("body"));
      replaceReturn = old.replaceReturn;
      return false;
    },

    visitReturnStatement(path) {
      if (replaceReturn) {
        assert(!inLoop); // could use labeled continue if this ever fires
        const { argument, comments } = path.node;
        if (argument === null) {
          const s = b.continueStatement();
          s.comments = comments;
          path.replace(s);
        } else {
          const s = b.expressionStatement(argument);
          s.comments = comments;
          path.replace(s, b.continueStatement());
        }
        return false;
      }
      this.traverse(path);
    },

    visitWhileStatement: visitLoop("test"),
  });

  if (changed) {
    console.log("Writing", file);
    fs.writeFileSync(file, recast.print(ast).code, { encoding: "utf8" });
  }
}

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-07 14:09:47 -08:00
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Steve Howell 4a03f91daa search: Retrofit recent changes to pills code.
This change makes these two functions more alike:

    - get_search_result
    - get_search_result_legacy

To test the UI modify zerver/views/home.py by
replacing `settings.SEARCH_PILLS_ENABLED` with
`True`.  I only did a quick sanity check, since
any bugs with the new system are more likely due
to bitrot than any changes I have made here.

The history is this:

    Tim cloned the code (before the smaller
    helpers were extracted):

        db4f6e278f

    In 8b153f6452
    Shubham removed get_operator_subset_suggestions but
    accidentally left a `concat` statement in that got
    misapplied to the previous suggestions:

    -    suggestions = get_operator_subset_suggestions(operators);
         result = result.concat(suggestions);

    The error there was carried over in some recent changes,
    but this commit fixes that strangeness.

    In 73e4f3b3fa
    Shubham made this change, which makes sense only for
    pills, and this code remains intact.

        -    if (operators.length > 0) {
        -        last = operators.slice(-1)[0];
        +    if (query_operators.length > 0) {
        +        last = query_operators.slice(-1)[0];
        +    } else {
        +        // If query_operators = [] then last will remain
        +        // {operator: '', operand: '', negated: false}; from above.
        +        // `last` has not yet been added to operators/query_operators.
        +        // The code below adds last to operators/query_operators
        +        operators.push(last);
        +        query_operators.push(last);
             }

    Mohit made a couple changes to both old and new.

    Anders made a couple non-substantive changes related to
    the ES6 migration.

    Steve (me) made several structural changes to the code.  For
    some of them I only changed the legacy code, not the pills
    code.  I didn't fix Shubham's mistake until this change.

Now the two functions should look similar except in the places
where they are intentionally different.  I also added a comment
explaining the get_operator_subset_suggestions difference.

Fixes #13609
2020-01-08 14:02:34 -08:00
Steve Howell 26168eaa98 search: Optimize search bar suggestions for large realms.
We only ever show 3 or 4 people in search suggestions
(possibly w/a couple variations, like pm-with/sender/etc.),
so we can try to search a smaller subset of people
before going through the entire realm.

We use message_store.user_ids() for this, since you
typically want to search messages for people that
have sent messages recently, and we already sort
based on PM conversations.
2020-01-04 12:58:00 -08:00
Steve Howell 32a1ef20d1 minor: Extract helper for search tests. 2019-12-28 05:43:04 -05:00
Anders Kaseorg 28f3dfa284 js: Automatically convert var to let and const in most files.
This commit was originally automatically generated using `tools/lint
--only=eslint --fix`.  It was then modified by tabbott to contain only
changes to a set of files that are unlikely to result in significant
merge conflicts with any open pull request, excluding about 20 files.
His plan is to merge the remaining changes with more precise care,
potentially involving merging parts of conflicting pull requests
before running the `eslint --fix` operation.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-03 12:42:39 -08:00
Mohit Gupta e5482adec0 search: Add streams:public to search entire history of public streams.
Add ability to search entire message history of all public streams at
once. It includes all subscibed, non subscribed public streams messages
and even historical public stream messages sent before user had joined
an organization or stream.

Fixes #8859.
2019-08-22 13:40:49 -07:00
Anders Kaseorg fb3fac1d96 zjsunit: Add make_handlebars abstraction.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-12 21:11:14 -07:00
Mohit Gupta 487861554f search: Fix searching and search suggestion in Group PM.
Fix the .get_suggestions and .get_suggestions_legacy
to correctly handle search terms in group PM and treat
it as search term by not concatenating it at end of pm-with
email list operand.
2019-06-28 12:45:37 -07:00
Pragati Agrawal 69ec96b63e search suggestion: Hide email under hidden email-address-visibility cases.
This commit hides the email address from the search bar for the email
hidden cases.
2019-05-30 16:22:54 -07:00
Shubham Padia 73e4f3b3fa search: Do not display `All messages` suggestion if bar not empty.
Previously `All messages` was displayed irrespective of the existing
pills. Now the suggestion is displayed only if no pills are present
2018-07-23 11:29:10 -07:00
Shubham Padia 8b153f6452 search: Validate suggestion against existing operators.
After adding search pills, suggestions were based only on the
current input and no validation against the existing pills was done.
operator_subset_suggestions have been removed. Default suggestions
for base_operators have also been removed.
Handle multiple operators:
if `is:starred stream:Ver` was typed without selecting the typeahead
or pressing enter in between i.e search pill for is:starred has not yet
been added, then the description of `is:starred` will act as a prefix
in every suggestion.
Also makes changes re-enabling person suggestions for names with spaces.
2018-07-23 11:29:10 -07:00
Tim Abbott 5c01ae87a0 search: Duplicate search node tests for pills.
This forks off search_legacy.js and search_suggestion_legacy.js so
that we can continue running automated tests against the legacy search
code while we develop the input pills feature.
2018-07-23 11:29:10 -07:00
Shubham Padia 1364971a17 search: Extract phrase_match to common.js.
Extracts phrase_match to common.js so it can be used by other
components like integrations search.
2018-06-26 20:38:47 -04:00
Shubham Padia 5d750a5d44 search: Respect spaces in person name in new suggestions only.
Fixes #6515.
New suggestions for `sender:King ha` will respect spaces and the new
suggestion will be `Sent by King Hamlet <email>` instead of `Sent by King,
search for ha`. But if first term of sender operand is a valid user email,
tokens will be seperated by spaces. e.g `sender:hamlet@zulip.com abc`
will show `Sent by King Hamlet <email>, search for abc`.
2018-05-25 09:03:27 -07:00
Shubham Padia e5b3fb332b search: Do not show default suggestion for `is` operator.
Fixes #9492.
Default suggestion e.g `abc messages` as a suggestion for `is:abc`
is not shown in a new suggestion. But if the is operator is already
present before any other operator, the default message text will be
used. e.g `is:abc sender:abc@zulipchat.com` will have all the suggestions
with the prefix `abc messages, sent by abc@zulipchat.com`.
2018-05-22 18:16:02 +05:30
Shubham Padia 4a78c00397 search: Rank "messages sent by" above "PMs with".
Fixes #9313.
2018-05-21 22:38:57 -07:00
Shubham Padia 17d4908667 search: Show negated suggestions for sent by me filters. 2018-05-21 22:38:14 -07:00
Shubham Padia d5976e9501 search: Show negated suggestions for stream filters.
Fixes #9461.
Adds negated suggestions for stream filters when the query is negated
which were previously being returned empty.
2018-05-21 22:38:14 -07:00
Shubham Padia 3d545abf9d search: Remove redundant get_containing_suggestions function.
`get_containing_suggestions` was used to get the operand suggestions
for the `has` operator. `get_special_filter_suggestions` is now used
to get both the operand and operator suggestions for `has`.
2018-05-21 22:38:14 -07:00
Shubham Padia 547f2ccae6 search: Show negated suggestions for category wise filters.
Partially fixes #9461.
Negated suggestion for both operand and operators are handle in
get_special_filter_suggestions. A bug is get_operator_suggestions
causing the removal of `-` symbol from the operand was also fixed.
2018-05-21 22:38:14 -07:00
Shubham Padia d478c6399c search: Show category-wise operator suggestions for has operator.
When suggesting operators to chose, category wise suggestions are
shown instead of a single default suggestion. e.g suggestions for
all the categories of has operator will be show instead of `Messages
with one or more` suggestion which did not make sense.
2018-05-18 14:18:26 -07:00
Shubham Padia 2945062b79 search: Do not show default suggestion for `has` operator.
Fixes #9384.
Default suggestion e.g `messages with one or more abc` as a suggestion
for `has:abc` is not shown in a new suggestion. But if the has operator
is already present before any other operator, the default message text
will be used. e.g `has:abc sender:abc@zulipchat.com` will have all the
suggestions with the prefix `messages with one or more abc, sent by
abc@zulipchat.com`.
2018-05-18 14:18:26 -07:00
Steve Howell 0dea143fcf Use unread data in topic_data.get_recent_names().
Fixes #9347.
2018-05-15 14:25:21 -07:00
Steve Howell 42435db492 Add run_test helper for individual tests.
This run_test helper sets up a convention that allows
us to give really short tracebacks for errors, and
eventually we can have more control over running
individual tests.  (The latter goal has some
complications, since we often intentionally leak
setup in tests.)
2018-05-15 08:24:44 -07:00
Tim Abbott 6e149a7594 lint: Add JS indentation eslint rules for node_tests.
The only difference between this as the main project's lint rules is
that we dont have the OuterIIFE setting.
2018-05-06 19:35:18 -07:00
Tim Abbott 65b9d9e0f3 CVE-2018-9990: Fix XSS issue with stream names in topic typeahead.
Zulip's search typeahead had a security bug, where when autocompleting
a specially crafted stream name, and then hitting space, code within
the stream name would be executed.

Zulip was doing HTML escaping correctly in the main code path using
Filter.describe to describe a narrow, but the escaping function was
not called in a few parallel code paths.  We fix this in a way that
should protect all of these code paths, by making Filter.describe
return properly escaped HTML, rather than depending on its callers to
do so.

Thanks to w2w for reporting this issue.
2018-04-12 09:46:54 -07:00
Rohitt Vashishtha b22e8dc2b7 lint: Replace 'return undefined;' with 'return;'.
Also adds a custom rule to eslint. Since the recommended way of extending
eslint is to create plugins as standalone npm packages, the separate rule
is published as 'eslint-plugins-empty-returns'.

Fixes #8669.
2018-03-13 08:22:42 -04:00
Steve Howell 19ca4bff82 node tests: Use zrequire in search_suggestion.js. 2017-11-08 12:24:17 -08:00
Cory Lynch a90eda0154 Add 'group-pm-with' to operator autocomplete.
As you're typing this new keyword, it should be suggested
via get_operator_suggestions. Added relevant test.
2017-09-24 13:56:33 -04:00
Abhijeet Kaur 2bf3324395 frontend: Add ability to search by "group-pm-with" search operator.
This allows user to view all group private conversation messages
with a specific user. That is, it views all the the group private
messages from groups which include the given user.

Add search suggestion for group-pm-with. Add operator name
and description in "Search operators" tab.

Add change in tab name to "Group Messages" when using this operator.
Add frontend_tests for group-pm-with search operator.

Fixes: #3882.
2017-09-24 11:58:48 -04:00
Steve Howell 5a8bccfe08 topic_data.js: Refactor topic history internals.
This commit introduces a per-stream topic_history class
inside of topic_data.js to better encapsulate how we store topic
history.

To the callers, nothing changes here.  (Some of our non-black-box
node tests change their way of setting up data, though, since the
internal data structures are different.)

The new class has the following improvements:

    * We use message_id instead of timestamp as our sorting key.
      (We could have done this in a prep commit, but it wouldn't
      have made the diff much cleaner here.)

    * We use a dictionary instead of a sorted list to store the
      data, so that writes are O(1) instead of O(NlogN).  Reads
      now do sorts, so they're O(NlogN) instead of O(N), but reads
      are fairly infrequent.  (The main goal here isn't actually
      performance, but instead it just simplifies the
      implementation.)

    * We isolate `topic_history` from the format of the messages.
      This prepares us for upcoming changes where updates to the
      data structure may come from topic history queries as well
      as messages.

    * We split out the message-add path from the message-remove
      path.  This prepares us to eventually get rid of the "count"
      mechanism that is kind of fragile and which has to be
      bypassed for historical topics.
2017-07-27 14:26:22 -07:00
Steve Howell bc0761b22b Extract topic_data.js.
This new module tracks the recent topic names for any given
stream.

The code was pulled over almost verbatim from stream_data.js,
with minor renames to the function names.

We introduced a minor one-line function called stream_has_topics.
2017-07-27 14:26:22 -07:00
Steve Howell b95e785335 recent topics: Extract get_recent_topic_names().
We now have all of our callers into recent_topics code just
receive a list of topic names from get_recent_topic_names().

This is more encapsulated than handing off tiny little
structures to the three callers, two of whom immediately
mapped the objects to names, and one of whom needlessly
used the now defunct name canon_subject field.

The consolidation here removes some "subject" references, and
now all lookup are by stream id, not stream name.

The diff here is a bit daunting, but it's mostly simplification
of tests and calling code.  Two of the callers now need to look
up stream ids, but they are otherwise streamlined.

The main change here is to stream_data.js, and we replace the
`canon_subject` and `subject` fields with `name`.
2017-07-27 14:26:22 -07:00
Cory Lynch a607130cae search_suggestion: Add tests for query with spaces.
Specifically, this checks to make sure that if you
surround an operand with quotes, having spaces inside
is permitted. Also, an extra space after the operator is
also permitted.
2017-07-14 17:33:09 -07:00
Cory Lynch 1dba70ca0d search_suggestion: Fix highlighting for multi-word streams.
I deleted a test case that involved a highlighted stream, but
the query was empty. This produces kind of a weird result with
typeahead_helper.highlight_with_escaping, but this function already
has coverage in node_tests/typeahead_helper.js, so the check here
was essentially redundant anyway. Specifically, the highlighter
wraps every character individually with <strong>, and looks really
messy in html.
2017-07-14 17:33:09 -07:00
Cory Lynch 4f8a2d0c57 filter.js: Make from/sender descriptions more concise. 2017-07-07 20:54:54 -07:00
Cory Lynch 2ce7a20237 search_suggestion: Support autocompleting operator names.
Fixes #5399
2017-07-07 20:54:54 -07:00
Cory Lynch 75617d07de search_suggestions: Prioritize recent huddles.
Fixes #5612. What this specifically does is that if you are
typing a group PM, this logic iterates through the possible
search suggestions for the next autocomplete. If that suggestion
contains a group PM that already exists, then prioritize it with
the most recent one on top.
2017-07-07 12:12:30 -07:00
Cory Lynch 7efba29b0c search_suggestions: Hide own name in huddle suggestions. 2017-07-07 12:12:30 -07:00
Cory Lynch 8abbb6d781 search: Remove in:all token from suggestions. 2017-07-01 12:19:52 -04:00
Cory Lynch 090d7487cf search_suggestion: Use person matching function from people.js.
Instead of having a custom (duplicate) matching function in
search suggestion, it was refactored to use the function in
people.js. This also gets the diacritic-ignoring feature
of the function in people.js.

Fixes #5315.
2017-06-23 08:29:20 -04:00
Cory Lynch 58a2ffbe1d Add search suggestion support for is:unread. 2017-06-19 06:51:13 -04:00
Cory Lynch 11a7a476d7 search_suggestion: Add negation support for 'has' operator. 2017-06-15 02:32:00 -07:00