Commit Graph

45734 Commits

Author SHA1 Message Date
Steve Howell 8ea320812f user exports: Chunkify messages in sorted order.
This accomplishes a few things:

    * It extracts `chunkify` rather than having us
      clumsily track chunking-related stuff in a
      big loop that is doing other stuff.

    * It makes it so that all message ids
      in message-000001.json < message-000002.json.

    * It makes it easier for us to customize
      the messages we send to a single user
      (coming soon).

BTW we probably have a slicker version of chunkify
somewhere in our codebase, but I couldn't remember
where.
2021-12-09 12:22:34 -08:00
Steve Howell 2a73964e16 user export: Add reactions.
We may eventually try to attach these to the messages
in the message-NNNNNN.json files, but for now they're
fine in user.json.
2021-12-09 12:22:34 -08:00
Eeshan Garg 3bab91079f external links: Migrate the rest of /developer-community links.
We recently changed /developer-community to /development-community.
Now that this change is in production, we can also migrate the
external links in our ReadTheDocs documentation.
2021-12-09 12:14:26 -08:00
Archit Hadge 25ccd4a7db settings: Change "N minutes" options to Custom.
We change the various "Up to N minutes" settings option labels to
"Custom", since the N is a little too mathy for some users.

Since the new prompts for the value of N are longer, we need to move
those prompts to the next line. Mainly this means switching from
`dependent-inline-block` to `dependent-block`, but we also need to
move the block out of the containing input-group for the CSS to be
happy.

Substantially rewritten by tabbott to use CSS for positioning and
change the message deleting copy of this issue.

Fixes #20177.
2021-12-09 11:52:27 -08:00
Swati Bhageria 2d766f3e78 message list: Display MOVED for messages with only topic edits.
This avoids the somewhat confusing visuals of showing messages as
EDITED where the content had not been changed, which also obscured
situations where a message had both been edited and moved.

It's possible we could do better with some sort of fancier block-move
visual styling, but it's a bit tricky to do well given that we support
moving multiple messages at once.

Fixes #20451.
2021-12-09 10:50:01 -08:00
Mateusz Mandera 93e18fe289 migrations: Remove disallowed characters from topics.
Following b3c58f454f, we want to clean up
old topics that may contain the disallowed characters. The Message table
is large, so we go in batches, making sure we limit topic fetches and
UPDATE query to no more than BATCH_SIZE Message rows per query.
2021-12-09 09:51:06 -08:00
Kevin 1d54b383bd compose: Fix keyboard UI with new global time widget.
Fixes #20130
2021-12-09 09:00:33 -08:00
Alya Abbott e70ec964dc portico: Add use case buttons to /why-zulip and /hello. 2021-12-09 08:58:45 -08:00
Aman Agrawal c35ab98d03 hello: Move call to action button style to a specific class. 2021-12-09 08:58:45 -08:00
Steve Howell f810833df5 export: Improve export_usermessages_batch.
We no longer jankily read our input file into
an "output" variable.  Instead, we do things
in a type-safe way.
2021-12-09 08:36:40 -08:00
Steve Howell 5c1e8cb8dc mypy: Add MessagePartial TypedDict. 2021-12-09 08:36:40 -08:00
Steve Howell 09c57a3f9f export: Log more consistently and sort ids.
Now all file writes go through our three
helper functions, and we consistently
write a single log message after the file
gets written.

I killed off write_message_exports, since
all but one of its callers can call
write_table_data, which automatically
sorts data.  In particular, our Message
and UserMessage data will now be sorted
by ids.
2021-12-09 08:36:40 -08:00
Alex Vandiver 053682964e puppet: Only fetch from running hosts in Grafana ec2 discovery. 2021-12-09 08:12:03 -08:00
Tim Abbott 36f03ccd5c message_events: Fix edit history live update when moving topics.
As noted in the TODO that we delete with this commit, we never
implemented live-updated for edit history when moving a message to
another topic.

Implementing this involves somewhat ugly copy-paste of the logic for a
content edit, but structurally is pretty simple.

It also makes #20451 much more visible.
2021-12-08 21:35:44 -08:00
Alex Vandiver 291f688678 puppet: Use zulip::external_dep for grafana, template config.
Templating the config ensures that the service is restarted when it is
upgraded.
2021-12-08 20:58:10 -08:00
Alex Vandiver 3eae429ab4 puppet: Upgrade Grafana to 8.3.1, for CVE-2021-43798. 2021-12-08 20:58:10 -08:00
Steve Howell 6ec49951c6 minor: Avoid creating intermediate list for message_ids.
This probably just postpones the list creation until
Django builds the "IN" query, but semantically it's
good to work in sets where we don't have any
meaningful ordering of the list that gets used.
2021-12-08 16:12:54 -08:00
Steve Howell f8ed099d3c export: Sort table data for most tables.
This affects most of our tables, but it excludes
table(s) like Message that go through kind of
unique codepaths.
2021-12-08 16:12:54 -08:00
Steve Howell a1d3f12e53 refactor: Extract write_table_data().
The immediate benefit of this is stronger mypy
checks (avoiding the ugly union caused by message
files).

The subsequent commit will add sorting.

We have test coverage on all these lines insofar
as if you comment out the lines, tests will
explode (i.e. more than superficial line
coverage).
2021-12-08 16:12:54 -08:00
Steve Howell c76ca2d0df export: Sort records.json files by path. 2021-12-08 16:12:54 -08:00
Steve Howell 2ef38e3d48 refactor: Extract write_records_json_file. 2021-12-08 16:12:54 -08:00
Steve Howell b79cfc19ab user export: Broaden query for RealmAuditLog.
We now check acting_user as well as modified_user
to see if a row pertains to our exported user.
2021-12-08 16:01:38 -08:00
Steve Howell 927b04368e minor: Use virtual_parent for custom fetchers.
The distinction here wasn't super meaningful
due to the way we order our "elif" statements,
but we want to reserver "normal_parent" for the
majority of use cases, where you simply tell
the Config what the "foreign_key" is.
2021-12-08 15:58:07 -08:00
Steve Howell 50120a9387 export: Remove config parameter for custom fetchers. 2021-12-08 15:58:07 -08:00
Steve Howell 54a3a423e5 mypy: Fix CustomFetch=Any hack. 2021-12-08 15:58:07 -08:00
Steve Howell 4128b52ac5 export: Rename custom fetchers. 2021-12-08 15:58:07 -08:00
Steve Howell a2c4931316 exports: Use realm for RealmAuditLog in realm exports.
For realm-wide exports, there is no reason to query
inefficiently against a list of modified users.

We move the Config out of the common child configs.
2021-12-08 15:58:07 -08:00
Steve Howell 8dd3c1038f exports: Rename parent_key to include_rows.
Even though Django usually treats foo__in
and foo_id__in identically for filters where
foo is a ForeignKey type, we want to insist
on somewhat more consistent syntax, because
we have the odd combo of type and type_id
in Recipient, where type_id is kinda like a
foreign key, but not a ForeignKey.

So we assert for now that all our include_rows
values end in "_id__in".
2021-12-08 15:58:07 -08:00
Steve Howell 02207f47d5 minor: Move code blocks to be alphabetical. 2021-12-08 15:58:07 -08:00
Steve Howell aae9f1b6f5 export: Make Config errors more clear. 2021-12-08 15:58:07 -08:00
Alex Vandiver 939d2e2705 scripts: Only stop/start existing tornado processes.
Stopping both `zulip-tornado` and `zulip-tornado:*` causes errors on
deploys with tornado sharding, as the plain `zulip-tornado` service
does not exist.

Pass `zulip-tornado:*`, which matches both plain `zulip-tornado`, as
well as the sharded `zulip-tornado:zulip-tornado-port-9800` cases.
2021-12-08 14:06:06 -08:00
Nikhil Maske 091772b534 hotspots: Remove intro_reply hotspot.
Zulip shows two guides on How to reply, first one by
the welcome bot and second one is intro_reply hotspot.
To simply and avoid redundancy, intro_reply hotspot is
removed.

Fixes #20482.
2021-12-07 21:55:59 -08:00
Sahil Batra 116a0f6f9d stream_settings: Sort subscriber list by name always.
This commit changes the behavior of subscriber list to
always be sorted by name instead of sorting them by email
when emails are accessible.

This change is fine because we will be using user-level
email address visibility and in that case the email of
some users will be visible and email of some will be not.
2021-12-07 21:44:40 -08:00
Sahil Batra 1ce869f16c settings_users: Fix comment in bot_info function. 2021-12-07 21:42:24 -08:00
Sahil Batra a1a5a1643f settings: Remove unused show_email parameter.
We show "Email" column heading always in users list and
subscriber list irrespective of the email-address visibility
setting after 46660e5, so we do need to pass show_email
parameter to render_admin_tab and render_stream_settings.
2021-12-07 21:42:23 -08:00
Nipunn Koorapati 0ca49bc93a emoji reactions: Order reactions query results by id.
Force postgres to give reactions in ID order - which
is generally chronological order. Results in frontend
displaying reactions in said order.

Fixes #20060.
2021-12-07 15:02:46 -08:00
Eeshan Garg 3714a30e63 stream notifications: Add helper for silent user mention syntax.
In many of our stream notification messages, we make use of the
same silent user mention syntax, the template for which was always
hardcoded. This commit adds a helper function that all relevant
callers can call to get the right syntax when mentioning users.

Thanks to Tim Abbott for this suggestion!
2021-12-07 14:53:50 -08:00
Eeshan Garg 8ebe05f644 streams: Add RealmAuditLog entry for message retention updates. 2021-12-07 14:53:50 -08:00
Eeshan Garg d2901892e2 streams: Add notifications for message retention policy updates.
This is a part of #20289.
2021-12-07 14:53:50 -08:00
Tim Abbott 73d503995a scripts: Fix running compare-settings-to-template from any CWD.
This matches the number of dirname() calls for other files in its
directory.

Fixes #20489.
2021-12-07 14:45:53 -08:00
Tim Abbott 48dccfd37a css: Fix night theme radio inputs.
Radio inputs that are not selected are technically independent
:read-only inputs, not a single input with multiple values; this
results in this selector for read-only inputs not behaving as
expected.

Fixes #20221.
2021-12-07 14:32:37 -08:00
S-Abhishek 186d1a83e9 narrow_banner: Move empty narrow messages to handlebar templates.
Removed existing empty narrow divs from app/home.html and created
a new javascript module to dynamically load empty narrow messages
using handlebar template.

Fixes #18797
2021-12-07 13:38:48 -08:00
Steve Howell 6381c2e535 tests: Make sure import doesn't corrupt original realm.
The original intention of this was to prevent coding
errors with realm getters that don't, um, filter
on realm.

Unfortunately, you can still write a broken realm getter
that forgets to filter on realm, but which returns a
Set, and the new safeguards won't see any difference.

We could make all the getters return sorted lists
instead, but that's for another day.

This code does serve another purpose, which is to
prevet egregious bugs in the import itself.
2021-12-07 12:27:01 -08:00
Steve Howell fea659eacd tests: Extract get_getters. 2021-12-07 12:27:01 -08:00
Steve Howell 5803057589 tests: Make some helpers class-level.
This is somewhat tactical in nature. I want to
extract a huge chunk of code that minorly depends
on these helpers.
2021-12-07 12:27:01 -08:00
Steve Howell 29bd1e8bd3 tests: Avoid clutter within long list of getters.
The diff here is ugly, but to summarize:

    BEFORE IMPORT:
        define get_user_id
        define get_huddle_hashes

    AFTER IMPORT AND MAKING GETTERS:
        check realm id
        define assert_realm_values
        verify emoji codes
        check huddle hashes
2021-12-07 12:27:01 -08:00
Steve Howell 93761cd237 tests: Add getter decorator for import test. 2021-12-07 12:27:01 -08:00
Steve Howell 5892748c7b tests: Avoid lambdas in import test. 2021-12-07 12:27:01 -08:00
Steve Howell 54a6c82282 tests: Avoid equal flag for huddle hashes.
There's no need to complexify the codepath
for all the normal use cases.
2021-12-07 12:27:01 -08:00
Lauryn Menard 58438c362f documentation: Fix text wrapping and alignment of ordered lists.
Moves CSS rules that rely on list items in an ordered list being
wrapped in a `<p>` tag so that they apply to the list item itself.
Uses `position: absolute` to set the `::before` pseudo-element in
place and `position: relative` to adjust the list items so that they
do not overlap.

Ideally, when Safari supports the `content` property for `::marker`
pseudo-elements, this issue can be revisited.

Fixes #20440.
2021-12-07 11:53:14 -08:00