Commit Graph

34066 Commits

Author SHA1 Message Date
Tim Abbott 0dc7542363 populate_db: Fix cache flushing when rebuilding test database.
This fixes a similar problem to the last commit; we don't use
memcached with the test database, so we don't need to flush memcached
when rebuilding it.

(And if we try, we'll get exceptions trying to access the relevant
settings).
2020-01-13 18:22:55 -08:00
Tim Abbott 571ce2f5cb populate_db: Fix handling of memcached flushing.
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.

We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.

We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
2020-01-13 18:05:21 -08:00
Anders Kaseorg 699626f3cf flush-memcached: Use pylibmc.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:18 -08:00
Anders Kaseorg 1ce15fba9c clear_database: Respect MEMCACHED_LOCATION.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:15 -08:00
Anders Kaseorg 6749810c2e puppet: Fix zuli-redis.conf path typo.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:37:09 -08:00
Anders Kaseorg 79cae1e7e0 puppet: Delete legacy rediscleanup code.
It was added in commit 9afb1c7a71 from
before 1.4.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:37:09 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
Steve Howell 15e7f5828b performance: Improve sort_recipients.
The sort_recipients helper is used for many different
typeaheads, such as compose PMs, compose mentions,
and some settings-related code.

We now avoid unnecessary sorting steps in cases
where we have plenty of results in the top buckets
(such as users who match on prefix).

This change should not have any user-facing
implications.
2020-01-12 20:53:03 -08:00
Steve Howell 9830d0a4c7 typeahead: Remove get_person_or_user_group_matcher.
We no longer need this, because we now filter
persons and groups separately.
2020-01-12 20:52:59 -08:00
Steve Howell bbb8abf4c5 typeahead: Extract composebox_typeahead.get_person_suggestions.
This method is a bit complex, but I think it's
worthwhile to force PM autocompletes and mention
autocompletes through the same code path.

We also kill off this method:

    typeahead_helper.sort_people_and_user_groups
2020-01-12 20:52:57 -08:00
Steve Howell 8fb9820b40 typeahead: Extract filter_and_sort_mentions.
We want a bit more control over the logic here.

The end game here is to converge a bit with
the code for autocompleting PM recipients.
2020-01-12 20:52:53 -08:00
Steve Howell 01ac607486 Fix non-int user_ids in user popovers. 2020-01-12 20:51:38 -08:00
Mateusz Mandera d5ac1afce8 email_mirror: Check address usability in get_missed_message_address. 2020-01-12 20:43:51 -08:00
Mateusz Mandera 89046ea1a9 email_mirror: Give extract_and_validate a more descriptive name. 2020-01-12 11:30:18 -08:00
Mateusz Mandera 90a69ab24f email_mirror: Reuse exception messages in mirror_email_message. 2020-01-12 11:30:18 -08:00
Steve Howell 38ac8e9f3e Convert more stream_ids to ints.
This fixes some regressions from a recent
commit that might not have been deployed.

9f7be51ce8

Even if that change had been deployed, it
should not have been user-facing, but it
would have spammed us with blueslip errors
every time somebody used the stream/topic
popovers in the left sidebar.
2020-01-12 11:27:26 -08:00
Steve Howell eba22e8e25 refactor: Inline get_person_or_user_group_matcher.
There's no reason any more to have a single function
filter both persons and groups.  Instead, we just
filter each cohort with a more direct function.

This is a minor performance speedup (avoiding the
conditional in the loop), but I mostly wanted to
simplify the code.
2020-01-09 17:46:33 -08:00
Steve Howell 60a0234732 refactor: Call sort_recipients directly.
The sort_people_and_user_groups function's only
value-add over sort_recipients is to split out
groups and users, but the caller already had
them split out, so it was kinda silly to concatenate
them back together.

I doubt this was a dumb decision at the time; I think
it was probably a consequence of how bootstrap's
normal approach is kinda inflexible when you're
using typeahead to pull data from multiple sources.

I wanted to kill off sort_people_and_user_groups
completely, but the mention/silent_mention autocompletes
are still a bit awkward to refactor.
2020-01-09 17:46:33 -08:00
Steve Howell 95daa50954 pm_list: Simplify is_all_privates logic.
For historical reasons pm_list was handling just
one possible edge case of where is:private was
combined with other search terms, namely the
pm-with operator.

The code was correct in realizing the is:private
was redundant there, but now we handle that
upstream in Filter.fix_operators (see previous
commit).

Now we just look for any is:private term.
2020-01-09 17:46:03 -08:00
Steve Howell 1af41dd954 filter: Remove redundant is:private operators.
If we have a pm-with, then is:private is redundant
and just forces us to write confusing/verbose code
in various places.
2020-01-09 17:46:03 -08:00
Ray Kraesig 88aa18ba85 docs: link to more-currently-maintained fork of GitX
The well-known rowanj/gitx repository hasn't been updated since 2014.
Preferentially direct new contributors to gitx/gitx instead.

(We retain the rowanj repo as a fallback, since it has precompiled
releases available.)
2020-01-09 16:06:37 -08:00
Tim Abbott d8df1255d3 find_team: Send find team emails from the support address.
This is for consistency with the email's body, which claims replying
directly will work.
2020-01-08 21:55:34 -08:00
Mateusz Mandera 9f2b0c769f stream_recipient: Eliminate unnecessary queries.
We should take adventage of the recipient field being denormalized into
the Stream model. We don't need to make queries to figure out a stream's
recipient id, so we take advantage of that to eliminate some of
those redundant queries and simplify StreamRecipientMap.
2020-01-08 14:34:43 -08:00
Mateusz Mandera 786c235023 stream_recipient: Optimize query in populate_for_recipient_ids.
There's no reason to join with the Stream table, as Recipient.type_id is
the stream id.
2020-01-08 14:34:43 -08:00
Steve Howell 4a03f91daa search: Retrofit recent changes to pills code.
This change makes these two functions more alike:

    - get_search_result
    - get_search_result_legacy

To test the UI modify zerver/views/home.py by
replacing `settings.SEARCH_PILLS_ENABLED` with
`True`.  I only did a quick sanity check, since
any bugs with the new system are more likely due
to bitrot than any changes I have made here.

The history is this:

    Tim cloned the code (before the smaller
    helpers were extracted):

        db4f6e278f

    In 8b153f6452
    Shubham removed get_operator_subset_suggestions but
    accidentally left a `concat` statement in that got
    misapplied to the previous suggestions:

    -    suggestions = get_operator_subset_suggestions(operators);
         result = result.concat(suggestions);

    The error there was carried over in some recent changes,
    but this commit fixes that strangeness.

    In 73e4f3b3fa
    Shubham made this change, which makes sense only for
    pills, and this code remains intact.

        -    if (operators.length > 0) {
        -        last = operators.slice(-1)[0];
        +    if (query_operators.length > 0) {
        +        last = query_operators.slice(-1)[0];
        +    } else {
        +        // If query_operators = [] then last will remain
        +        // {operator: '', operand: '', negated: false}; from above.
        +        // `last` has not yet been added to operators/query_operators.
        +        // The code below adds last to operators/query_operators
        +        operators.push(last);
        +        query_operators.push(last);
             }

    Mohit made a couple changes to both old and new.

    Anders made a couple non-substantive changes related to
    the ES6 migration.

    Steve (me) made several structural changes to the code.  For
    some of them I only changed the legacy code, not the pills
    code.  I didn't fix Shubham's mistake until this change.

Now the two functions should look similar except in the places
where they are intentionally different.  I also added a comment
explaining the get_operator_subset_suggestions difference.

Fixes #13609
2020-01-08 14:02:34 -08:00
Tim Abbott 56946dbddb design: Fix missing rendered_markdown class on /me content.
There may be a deeper issue that various JavaScript logic expects
every message to have a `.message_content` element, but we definitely
should have the `.rendered_markdown` class on any markdown content.

Fixes #13634.
2020-01-08 13:46:27 -08:00
Steve Howell e1c3cd05df refactor: Improve names in sort_recipients.
The reason we use functions here will be clear
in the next change.

This is a "prefactor" commit that doesn't change
any user-facing behavior nor significantly change
performance.
2020-01-08 12:57:51 -08:00
Steve Howell 33af2c2ce1 minor: Avoid groups conditionals.
Instead of short-circuiting the groups code every
time, just make it an empty array as needed.

The benefits of this will be more obvious soon.
2020-01-08 12:57:51 -08:00
Steve Howell 583de13553 refactor: Extract partition helper.
A few reasons to extract it:

    - we can shorten lines (and not repeat query
      every time)

    - we can scope the big block comment explaining
      why util.prefix_sort is a strange name

    - the name is better (it's an O(N) operation that
      mostly partitions)

    - we may want to swap it out with a true partition
      function that's truly a partition (since the
      case checks done by prefix_sort are possibly
      either a non-feature or mostly overridden by
      the other sorts)
2020-01-08 12:57:47 -08:00
Steve Howell 25fa918f93 refactor: Extract sort_relevance helper. 2020-01-08 12:55:42 -08:00
Steve Howell f47f27d110 refactor: Use filter_taken_users in compose typeahead.
The composebox_typeahead code now gets people directly,
which will allow us to do some optimizations upcoming.
2020-01-08 12:55:42 -08:00
Steve Howell d258a27a79 refactor: Extract filter_taken_users.
We will take advantage of this in a few more
commits.
2020-01-08 12:55:42 -08:00
shubhamgupta2956 92817e147d webhooks: Remove TrelloWebhookException classes.
This removes zerver/webhooks/trello/view/exceptions.py, which
contained legacy Trello webhook exception related classes.  We replace
them with UnexpectedWebhookEventType, which results in our standard
exception handling for unknown event types running (avoiding too-high
priority error logging).

Fixes #13467.
2020-01-07 18:02:40 -08:00
Anders Kaseorg 5526af32f3 puppet: Switch double quoted strings to single quoted.
Resolves these warnings from puppet-lint.

puppet-lint| puppet/zulip/manifests/app_frontend_base.pp - WARNING: double quoted string containing no variables on line 14
puppet-lint| puppet/zulip/manifests/app_frontend_base.pp - WARNING: double quoted string containing no variables on line 19

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-07 15:10:17 -08:00
rht cd3907648d prod install: Use ID_LIKE to help select os family. 2020-01-07 13:25:25 -08:00
rht bc94e8e815 prod install: Use /etc/os-release for Ubuntu/Debian to get os_id, os_version_id. 2020-01-07 13:25:25 -08:00
rht 70dfb423e4 puppet: Specify CentOS specific path to ca certificates for nginx. 2020-01-07 13:25:25 -08:00
rht d5284b177e puppet: Convert memorysize_mb to integer depending on Puppet version. 2020-01-07 13:25:25 -08:00
rht 60563e9180 provision: Install postgresql-devel on CentOS as system dependency. 2020-01-07 13:25:25 -08:00
rht 9898c07e0d prod install: Add the CentOS version of the step to do dist-upgrade. 2020-01-07 13:25:25 -08:00
rht bf76696d67 prod install: Add the CentOS version of the step to install preparatory packages. 2020-01-07 13:25:25 -08:00
rht 6dd5dc32fc prod install: Add the CentOS version of the step to upgrade packages. 2020-01-07 13:25:25 -08:00
rht d88a7bbb42 prod install: Add the CentOS version of the step to update packages. 2020-01-07 13:25:25 -08:00
rht 49d7adb3cb prod install: Parse CentOS os identifications from /etc/os-release. 2020-01-07 13:25:25 -08:00
rht 771f6d213f prod install: Rename os_codename into os_version_id 2020-01-07 13:25:25 -08:00
rht dccfb0ebe9 puppet: Remove duplicate postgresql-client safepackage check on CentOS. 2020-01-07 13:25:25 -08:00
Hashir Sarwar 0cabacb8ab export: Fix data export parallelization.
This improves the approach of creating multiple parallel processes by
using subprocess.Popen() instead of run_parallel() and
subprocess.call() while exporting an organization's message
history.  This prevents forking twice for individual subprocess.

While this has some performance benefit, the main reason to fix this
is that it fixes an issue with the data export web UI introduced in
run_parallel forks exited).

Fixes #12904.
2020-01-07 13:23:18 -08:00
Mateusz Mandera b87cf22b33 email_mirror: Move send_to_mm_address code to process_missed_message.
process_missed_message did nothing other than calling
send_to_missed_message_address with the same arguments, so there's no
reason to have these as separate functions.
2020-01-07 13:03:32 -08:00
Mateusz Mandera c011d2c6d3 email_mirror: Migrate missed message addresses from redis to database.
Addresses point 1 of #13533.

MissedMessageEmailAddress objects get tied to the specific that was
missed by the user. A useful benefit of that is that email message sent
to that address will handle topic changes - if the message that was
missed gets its topic changed, the email response will get posted under
the new topic, while in the old model it would get posted under the
old topic, which could potentially be confusing.

Migrating redis data to this new model is a bit tricky, so the migration
code has comments explaining some of the compromises made there, and
test_migrations.py tests handling of the various possible cases that
could arise.
2020-01-07 13:03:22 -08:00
Mateusz Mandera 9077bbfefd models: Add MissedMessageEmailAddress class.
Preparatory commit for making the email mirror use the database instead
of redis for missed message addresses.

This model will represent missed message email addresses, which
currently have their data stored in redis.
The redis data will be converted and migrated into these models and
the email mirror will start using them in the main commit.
2020-01-07 12:46:55 -08:00