Commit Graph

34237 Commits

Author SHA1 Message Date
Steve Howell 752d6dc6df tools: Remove find-add-class tool.
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.

At this point the tool is just a little too noisy
to be helpful.  An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS.  I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.

I just don't think it's worth maintaining the tool
at this point.
2020-01-14 15:45:49 -08:00
Steve Howell 29e63c0417 Fix type errors in LazySet.
I think the only place that was broken is where
we copy users from streams.
2020-01-14 15:40:40 -08:00
Steve Howell b65138c83f minor: Make type conversion explicit. 2020-01-14 15:40:40 -08:00
Mateusz Mandera 0beae44081 email_mirror: Use .walk() to search all MIME parts for attachments.
Fixes #13416

We used to search only one level in depth through the MIME structure,
and thus would miss attachments that were nested deeper (which can
happen with some email clients). We can take advantage of message.walk()
to iterate through each MIME part.
2020-01-14 15:37:39 -08:00
Mateusz Mandera c579b6858e send_to_email_mirror: Fix loop setting recipient-like headers.
return in that loop was a bug, which would lead to the To: header not
being set even though data['recipient'] = str(message['To']) is being
run next, thus requiring the header. We can remove the return
statement and now the loop will overwrite all the potentially
troublesome headers.
2020-01-14 15:37:39 -08:00
Mateusz Mandera 1561d144e0 email_mirror: Insert a new line before attachment links. 2020-01-14 15:37:39 -08:00
Tim Abbott 8226573af6 default stream groups: Fix broken registration UI.
The default stream groups feature (#6693) was never fully implemented;
this fixes a key detail (the registration UI being broken).
2020-01-14 14:50:18 -08:00
Tim Abbott 4562949f43 default stream groups: Fix buggy LDAP behavior.
With LDAP authentication, we don't currently have a good way to
support the default stream groups feature.

The old behavior was just to assume a user select every default stream
group, which seems wrong; since we didn't prompt the user about these,
we should just ignore the feature.
2020-01-14 14:50:18 -08:00
Pragati Agrawal 0eafa48ca1 org settings: Fix error of wrong type of argument passed to InDict.has().
This fixes the error where we pass `user_id` of 'string' type as the
argument instead of 'integer' to `exports.get_person_from_user_id` which
further passes `user_id` to InDict.has() function which accepts integer
argument only.
2020-01-14 14:38:26 -08:00
Tim Abbott 80b9acd745 compose: Update some comments on private stream warnings. 2020-01-14 13:23:27 -08:00
Steve Howell c2af2c1fd1 refactor: Extract is_subscriber_subset().
Extracting the function makes it a bit easier to
test and use in a generic way.

Also, I wanted this to live in stream_data, so that
it's easier to find if we change how we model
subscriber data.

Finally, I use _.every to do the subset check
instead of `_.difference`, since _.difference
is actually N-squared:

  _.difference = restArguments(function(array, rest) {
    rest = flatten(rest, true, true);
    return _.filter(array, function(value){
      return !_.contains(rest, value);
    });
  });

And we don't actually want to build a list only
to check that it's zero vs. nonzero length.

We now do this, which short circuits as soon
as it finds any key that is only in sub1:

    return _.every(sub1.subscribers.keys(), (key) => {
        return sub2_set.has(key);
    });
2020-01-14 13:19:49 -08:00
Steve Howell 34b21bc0ee refactor: Use is_broadcast flag for mention check.
I also clean up the noop tests here, which were
actually redundant (all three cases were short
circuiting on the "everyone" mention).
2020-01-14 13:19:49 -08:00
Steve Howell e638361728 minor: Move unit test to module scope.
This test is really no longer an "event" test.
2020-01-14 13:19:49 -08:00
Steve Howell 593049d551 compose: Extract warn_if_mentioning_unsubscribed_user.
First, there are no more convoluted signals.

We also simplify the parameter to just the "mentioned"
object corresponding to either a user or a broadcast
mention.

For the user group scenario, this has always been dead
code, which you only realized when you got to the comment
at the bottom.  Now we actually do nothing.
And I moved the relevant commment to the
the typeahead code (with new wording).

I also moved the is_silent check to the caller.  I don't
feel too strongly about that either way. It's kind of silly
to call a function only to give that function an additional
responsibility to worry about.  On the other hand, I see
the logic of that function enforcing everything.  I went
with the former for now.

Arguably we should have a warning for silent mentions,
since doing a silent mention of somebody not on a stream
is a good indication of a typo.  I do understand the use
case, but the user can always ignore the warning.  Anyway,
we have decent test coverage on this.
2020-01-14 13:19:45 -08:00
Steve Howell b91a19df43 refactor: Extract warn_if_private_stream_is_linked.
This isn't really an extraction; it's more giving
a name to an anonymous function and moving it to
higher module scope.

We convert this to an ordinary function call, which
allows us to move it out of intialize().

Since there's just one simple parameter now (linked_stream),
we can avoid some error checking.

We also avoid the comment that describes the function,
since it now has a name.

And then one minor tweak is to do the inexpensive
`invite_only` higher in the function.  This will be
a nice speedup when you link to really large public
streams.

The unit tests are also a bit easier to read now--less
setup and more explicit names.
2020-01-14 13:13:48 -08:00
Tim Abbott c10cc24ee8 python: Sort webhooks imports with isort. 2020-01-14 13:07:47 -08:00
Tim Abbott 8e7ce7cc79 python: Sort migrations/management command imports with isort.
This is a preparatory commit for using isort for sorting all of our
imports, merging changes to files where we can easily review the
changes as something we're happy with.

These are also files with relatively little active development, which
means we don't expect much merge conflict risk from these changes.
2020-01-14 13:07:47 -08:00
Steve Howell 9f8eccdbbf hash_util: Simplify pm_with_uri.
This is easier to read and faster, because it
avoids some unnecessary encoding on the pm-with
part, plus just a lot of extra logic that amounts
to just appending the slug.

Performance for this function is relevant because it is used
for every user every time we rerender the right sidebar.
2020-01-14 12:39:17 -08:00
Steve Howell 9f590889b7 refactor: Use a Set for away_user_ids. 2020-01-14 17:52:25 +00:00
Steve Howell d8554e085c IntDict: Use IntDict for user_info in user_status. 2020-01-14 17:52:16 +00:00
Tlazypanda 30ee0c2a49 invitations: Improve experience around reactivating users.
Previously, if you tried to invite a user whose account had been
deactivated, we didn't provide a clear path forward for reactivating
the users, which was confusing.

We fix this by plumbing through to the frontend the information that
there is an existing user account with that email address in this
organization, but that it's deactivated.  For administrators, we
provide a link for how to reactivate the user.

Fixes #8144.
2020-01-13 18:30:51 -08:00
Tim Abbott 0dc7542363 populate_db: Fix cache flushing when rebuilding test database.
This fixes a similar problem to the last commit; we don't use
memcached with the test database, so we don't need to flush memcached
when rebuilding it.

(And if we try, we'll get exceptions trying to access the relevant
settings).
2020-01-13 18:22:55 -08:00
Tim Abbott 571ce2f5cb populate_db: Fix handling of memcached flushing.
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.

We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.

We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
2020-01-13 18:05:21 -08:00
Anders Kaseorg 699626f3cf flush-memcached: Use pylibmc.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:18 -08:00
Anders Kaseorg 1ce15fba9c clear_database: Respect MEMCACHED_LOCATION.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:15 -08:00
Anders Kaseorg 6749810c2e puppet: Fix zuli-redis.conf path typo.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:37:09 -08:00
Anders Kaseorg 79cae1e7e0 puppet: Delete legacy rediscleanup code.
It was added in commit 9afb1c7a71 from
before 1.4.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:37:09 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
Steve Howell 15e7f5828b performance: Improve sort_recipients.
The sort_recipients helper is used for many different
typeaheads, such as compose PMs, compose mentions,
and some settings-related code.

We now avoid unnecessary sorting steps in cases
where we have plenty of results in the top buckets
(such as users who match on prefix).

This change should not have any user-facing
implications.
2020-01-12 20:53:03 -08:00
Steve Howell 9830d0a4c7 typeahead: Remove get_person_or_user_group_matcher.
We no longer need this, because we now filter
persons and groups separately.
2020-01-12 20:52:59 -08:00
Steve Howell bbb8abf4c5 typeahead: Extract composebox_typeahead.get_person_suggestions.
This method is a bit complex, but I think it's
worthwhile to force PM autocompletes and mention
autocompletes through the same code path.

We also kill off this method:

    typeahead_helper.sort_people_and_user_groups
2020-01-12 20:52:57 -08:00
Steve Howell 8fb9820b40 typeahead: Extract filter_and_sort_mentions.
We want a bit more control over the logic here.

The end game here is to converge a bit with
the code for autocompleting PM recipients.
2020-01-12 20:52:53 -08:00
Steve Howell 01ac607486 Fix non-int user_ids in user popovers. 2020-01-12 20:51:38 -08:00
Mateusz Mandera d5ac1afce8 email_mirror: Check address usability in get_missed_message_address. 2020-01-12 20:43:51 -08:00
Mateusz Mandera 89046ea1a9 email_mirror: Give extract_and_validate a more descriptive name. 2020-01-12 11:30:18 -08:00
Mateusz Mandera 90a69ab24f email_mirror: Reuse exception messages in mirror_email_message. 2020-01-12 11:30:18 -08:00
Steve Howell 38ac8e9f3e Convert more stream_ids to ints.
This fixes some regressions from a recent
commit that might not have been deployed.

9f7be51ce8

Even if that change had been deployed, it
should not have been user-facing, but it
would have spammed us with blueslip errors
every time somebody used the stream/topic
popovers in the left sidebar.
2020-01-12 11:27:26 -08:00
Steve Howell eba22e8e25 refactor: Inline get_person_or_user_group_matcher.
There's no reason any more to have a single function
filter both persons and groups.  Instead, we just
filter each cohort with a more direct function.

This is a minor performance speedup (avoiding the
conditional in the loop), but I mostly wanted to
simplify the code.
2020-01-09 17:46:33 -08:00
Steve Howell 60a0234732 refactor: Call sort_recipients directly.
The sort_people_and_user_groups function's only
value-add over sort_recipients is to split out
groups and users, but the caller already had
them split out, so it was kinda silly to concatenate
them back together.

I doubt this was a dumb decision at the time; I think
it was probably a consequence of how bootstrap's
normal approach is kinda inflexible when you're
using typeahead to pull data from multiple sources.

I wanted to kill off sort_people_and_user_groups
completely, but the mention/silent_mention autocompletes
are still a bit awkward to refactor.
2020-01-09 17:46:33 -08:00
Steve Howell 95daa50954 pm_list: Simplify is_all_privates logic.
For historical reasons pm_list was handling just
one possible edge case of where is:private was
combined with other search terms, namely the
pm-with operator.

The code was correct in realizing the is:private
was redundant there, but now we handle that
upstream in Filter.fix_operators (see previous
commit).

Now we just look for any is:private term.
2020-01-09 17:46:03 -08:00
Steve Howell 1af41dd954 filter: Remove redundant is:private operators.
If we have a pm-with, then is:private is redundant
and just forces us to write confusing/verbose code
in various places.
2020-01-09 17:46:03 -08:00
Ray Kraesig 88aa18ba85 docs: link to more-currently-maintained fork of GitX
The well-known rowanj/gitx repository hasn't been updated since 2014.
Preferentially direct new contributors to gitx/gitx instead.

(We retain the rowanj repo as a fallback, since it has precompiled
releases available.)
2020-01-09 16:06:37 -08:00
Tim Abbott d8df1255d3 find_team: Send find team emails from the support address.
This is for consistency with the email's body, which claims replying
directly will work.
2020-01-08 21:55:34 -08:00
Mateusz Mandera 9f2b0c769f stream_recipient: Eliminate unnecessary queries.
We should take adventage of the recipient field being denormalized into
the Stream model. We don't need to make queries to figure out a stream's
recipient id, so we take advantage of that to eliminate some of
those redundant queries and simplify StreamRecipientMap.
2020-01-08 14:34:43 -08:00
Mateusz Mandera 786c235023 stream_recipient: Optimize query in populate_for_recipient_ids.
There's no reason to join with the Stream table, as Recipient.type_id is
the stream id.
2020-01-08 14:34:43 -08:00
Steve Howell 4a03f91daa search: Retrofit recent changes to pills code.
This change makes these two functions more alike:

    - get_search_result
    - get_search_result_legacy

To test the UI modify zerver/views/home.py by
replacing `settings.SEARCH_PILLS_ENABLED` with
`True`.  I only did a quick sanity check, since
any bugs with the new system are more likely due
to bitrot than any changes I have made here.

The history is this:

    Tim cloned the code (before the smaller
    helpers were extracted):

        db4f6e278f

    In 8b153f6452
    Shubham removed get_operator_subset_suggestions but
    accidentally left a `concat` statement in that got
    misapplied to the previous suggestions:

    -    suggestions = get_operator_subset_suggestions(operators);
         result = result.concat(suggestions);

    The error there was carried over in some recent changes,
    but this commit fixes that strangeness.

    In 73e4f3b3fa
    Shubham made this change, which makes sense only for
    pills, and this code remains intact.

        -    if (operators.length > 0) {
        -        last = operators.slice(-1)[0];
        +    if (query_operators.length > 0) {
        +        last = query_operators.slice(-1)[0];
        +    } else {
        +        // If query_operators = [] then last will remain
        +        // {operator: '', operand: '', negated: false}; from above.
        +        // `last` has not yet been added to operators/query_operators.
        +        // The code below adds last to operators/query_operators
        +        operators.push(last);
        +        query_operators.push(last);
             }

    Mohit made a couple changes to both old and new.

    Anders made a couple non-substantive changes related to
    the ES6 migration.

    Steve (me) made several structural changes to the code.  For
    some of them I only changed the legacy code, not the pills
    code.  I didn't fix Shubham's mistake until this change.

Now the two functions should look similar except in the places
where they are intentionally different.  I also added a comment
explaining the get_operator_subset_suggestions difference.

Fixes #13609
2020-01-08 14:02:34 -08:00
Tim Abbott 56946dbddb design: Fix missing rendered_markdown class on /me content.
There may be a deeper issue that various JavaScript logic expects
every message to have a `.message_content` element, but we definitely
should have the `.rendered_markdown` class on any markdown content.

Fixes #13634.
2020-01-08 13:46:27 -08:00
Steve Howell e1c3cd05df refactor: Improve names in sort_recipients.
The reason we use functions here will be clear
in the next change.

This is a "prefactor" commit that doesn't change
any user-facing behavior nor significantly change
performance.
2020-01-08 12:57:51 -08:00
Steve Howell 33af2c2ce1 minor: Avoid groups conditionals.
Instead of short-circuiting the groups code every
time, just make it an empty array as needed.

The benefits of this will be more obvious soon.
2020-01-08 12:57:51 -08:00
Steve Howell 583de13553 refactor: Extract partition helper.
A few reasons to extract it:

    - we can shorten lines (and not repeat query
      every time)

    - we can scope the big block comment explaining
      why util.prefix_sort is a strange name

    - the name is better (it's an O(N) operation that
      mostly partitions)

    - we may want to swap it out with a true partition
      function that's truly a partition (since the
      case checks done by prefix_sort are possibly
      either a non-feature or mostly overridden by
      the other sorts)
2020-01-08 12:57:47 -08:00