Commit Graph

524 Commits

Author SHA1 Message Date
Anders Kaseorg 9d0340c7a8 rundjangoserver: Filter stdout instead of forking the code.
We failed to update this fork for the Django 3.2 upgrade.  Unfork it
so that’s not something we need to remember to do.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-24 18:54:47 -07:00
Anders Kaseorg d1aa68994a render_messages: Use JSON Lines format.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-20 13:12:16 -07:00
Abhijeet Prasad Bodas ec8a931761 message send: Pass individual parameters instead of single Dict.
This will allow for stronger type checking and better readability.
2021-05-20 11:06:19 -07:00
sophie200 fae2265e5d populate_db: Remove use of deleted pointer field.
Apparently, when we removed the pointer from our data model, we
neglected to delete this update code from populate_db.
2021-05-03 09:32:57 -07:00
Vishnu KS 7f3fc3423b audit log: Create audit log when a realm is created.
This is mainly useful in recording the user who created the realm,
when possible.
2021-04-30 09:25:11 -07:00
Anders Kaseorg e7ed907cf6 python: Convert deprecated Django ugettext alias to gettext.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Arun Sankar 146b32d63a test users: Add an escape char to a test username.
Changed the name of the test-user cordelia from `Cordelia Lear` to
`Cordelia, Lear's daughter`.

This change will enable us to test users with escape characters in
their names.

I also updated the Node, Puppeteer, Backend tests and Fixtures to
support this change.
2021-04-13 11:42:06 -07:00
Anders Kaseorg f0e655f1d8 request: Rename validator parameter of REQ to json_validator.
This makes it much more clear that this feature does JSON encoding,
which previously was only indicated in the documentation.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 14:13:06 -07:00
Anders Kaseorg 93d2ae8092 request: Remove redundant str_validator=check_string from REQ().
REQ(str_validator=check_string) is equivalent to the default behavior
of REQ().

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 14:13:03 -07:00
Abhijeet Prasad Bodas 2698acd738 management: Fix incorrect help text in compare_messages.
This was added, along with `render_messages`, in
93965a8e89.
2021-04-05 18:08:18 -07:00
Arun Sankar 93c88705cf default data: Add non ASCII and non BMP characters to stream names.
Added  emojis and Non ASCII characters to default
stream names and descriptions.

Added raw_emojis array under streams so that
we can pass --extra-streams argument without
--extra-users as it should be.
2021-04-02 22:00:07 -07:00
shanukun 0bf067b681 refactor: Make acting_user a mandatory kwarg for bulk_add_subscriptions. 2021-04-02 14:44:41 -07:00
Arun Sankar 954aced781 populate_db: Add non ascii and non bmp characters to Stream names.
Added non ascii and non bmp characters to stream names.

A Stream Name will now consist of a random stream name +
a number (to avoid name duplicates) + a 15% to contain
a emoji.
2021-03-31 13:00:56 -07:00
Arun Sankar b26b647b1d populate_db: Add non ASCII and non BMP characters to usernames.
Added non ASCII and non bmp characters to full name.

Created a new list for non_ascii_names and emojis
to store them explicitly.

A full name will now consist of first name +
(a non ASCII name or a plain middle name) + (a emoji
or a plain last name).

First name will not have any non ASCII or non bmp text
as it is also being used as email.
2021-03-31 13:00:56 -07:00
Mateusz Mandera f329878376 migrations: Subscription.is_user_active denormalization - step one.
This adds the is_user_active with the appropriate code for setting the
value correctly in the future. In the following commit a migration to
backfill the value for existing Subscriptions will be added.

To ensure correct user_profile.is_active handling also in tests, we
replace all direct .is_active mutation with calls to appropriate
functions.
2021-03-30 09:19:03 -07:00
tushar912 b220d29fed custom profile fields: Rename "CHOICE" to "SELECT" in backend.
Rename the "CHOICE" field to "SELECT" in backend. This is
done to improve readability as a prep for the upcoming
"SELECT_MULTIPLE" field.
2021-03-24 12:54:51 -07:00
Mateusz Mandera d91d3a05b9 tests: Use do_create_realm where possible.
Using do_create_realm should be preferred over manual creation where
possible, as it creates more realistic data.
2021-03-14 08:50:02 -07:00
Mateusz Mandera 51d7f24d20 actions: Remove realm argument to internal_send_stream_message.
The argument is redundant.
2021-02-23 15:26:47 -08:00
sahil839 6b5cf231a1 users: Add new user 'shiva' as realm moderator.
Note that at this point, it's not possible to create moderator users;
this just will make it easier to write tests for logic involving them
as we develop the feature.
2021-02-23 15:00:49 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Mateusz Mandera 90636d5e81 events: Fix bug in get_recent_conversations_recipient_id.
user_profile.id was confused for user_profile.recipient_id. These bugs
are particularly sneaky as they can go undetected by tests due to ids of
objects accidentally coinciding. We add a mitigation for this class of
mistakes by shifting the Recipient.id sequence in test db.

This was introduced in dda3ff41e1.
On the rare occasion where user_profile.id would coincide with
recipient_id passed to the function, we would return the wrong value.
That is, instead of correctly returning recipient_id, we would return
sender.recipient_id - recipient id of the sender of the message, thus
possibly returning user_profile.recipient_id (if user_profile is the
sender) - exactly the situation the function wanted to avoid
with the `if recipient_id == my_recipient_id:` if. Ultimately resulting
in incorrect/malformed data in
state['raw_recent_private_conversations'].
2021-02-09 17:45:34 -08:00
Tim Abbott 38ffaad325 django: Add custom runserver wrapper to limit startup logging.
This helps considerably in avoiding the `run-dev.py` startup output
confusing developers.
2020-12-20 12:11:16 -08:00
Anders Kaseorg 2ab0b3d4fc validator: Reject ISO 8601 dates missing leading zeros.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-15 16:36:50 -08:00
Steve Howell 99e725cbde populate_db: Simplify how we create reactions.
For 3000 messages and 400 users, this saved
about 30 seconds.

We only do two queries per batch of messages
now, and the algorithm is easier to analyze,
as it's just three nested loops.
2020-11-16 17:19:23 -08:00
Anders Kaseorg 31d0141a30 python: Close opened files.
Fixes various instances of ‘ResourceWarning: unclosed file’ with
python -Wd.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-26 12:31:30 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Steve Howell 9df9934ed6 refactor: Pass realm to bulk_add_subscriptions.
I think it's important that the callers understand
that bulk_add_subscriptions assumes all streams
are being created within a single realm, so I make
it an explicit parameter.

This may be overkill--I would also be happy if we
just included the assertions from this commit.
2020-10-13 11:28:17 -07:00
Alex Vandiver d47637fa40 queue: Set a max consume timeout with SIGALRM.
SIGALRM is the simplest way to set a specific maximum duration that
queue workers can take to handle a specific message.  This only works
in non-threaded environments, however, as signal handlers are
per-process, not per-thread.

The MAX_CONSUME_SECONDS is set quite high, at 10s -- the longest
average worker consume time is embed_links, which hovers near 1s.
Since just knowing the recent mean does not give much information[1],
it is difficult to know how much variance is expected.  As such, we
set the threshold to be such that only events which are significant
outliers will be timed out.  This can be tuned downwards as more
statistics are gathered on the runtime of the workers.

The exception to this is DeferredWorker, which deals with quite-long
requests, and thus has no enforceable SLO.

[1] https://www.autodesk.com/research/publications/same-stats-different-graphs
2020-10-06 17:26:14 -07:00
Alex Vandiver 8cf37a0d4b queue: Add a tool to profile no-op enqueue and dequeue actions. 2020-10-06 17:26:14 -07:00
sahil839 2e58e27926 populate_db: Set wildcard_mention_policy to show warning in test_suite.
We set wildcard_mention_policy in the test database so that we can
avoid future changes in mention puppeteer tests, as the default
membership of streams in the Zulip development organization is large
enough to prevent random users from using wildcard mentions.
2020-10-01 12:20:41 -07:00
sahil839 6c473ed75f message: Call build_message_send_dict from check_message.
We call build_message_send_dict from check_message instead of
do_send_messages.

This is a prep commit for adding a new setting for handling
wildcard mentions in large streams.
2020-09-29 17:18:04 -07:00
Wes Galbraith 9645959ac4 populate_db: Add emoji reactions to development environment database.
This change adds automated generated emoji reactions to the data in
the development environment's database.

Fixes part of #14991.
2020-09-23 16:10:37 -07:00
Anders Kaseorg 0228acf0f5 rest: Add rest_path shortcut for path with rest_dispatch.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:51:00 -07:00
Anders Kaseorg 5297e4a30a urls: Use unqualified imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:51:00 -07:00
Anders Kaseorg e70f2ae58d rest: Specify rest_dispatch handlers by function, not by string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:46:28 -07:00
Alex Vandiver e5f62d083e tornado: Merge the TORNADO_SERVER and TORNADO_PORTS configs.
Having both of these is confusing; TORNADO_SERVER is used only when
there is one TORNADO_PORT.  Its primary use is actually to be _unset_,
and signal that in-process handling is to be done.

Rename to USING_TORNADO, to parallel the existing USING_RABBITMQ, and
switch the places that used it for its contents to using
TORNADO_PORTS.
2020-09-21 15:36:16 -07:00
Anders Kaseorg a50fae89e2 python: Elide type=str from argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 1f2ac1962f python: Elide default=None for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Hashir Sarwar b885678881 push_notifications: Simplify `if device exists` checks. 2020-08-31 17:31:41 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg dbdf67301b memcached: Switch from pylibmc to python-binary-memcached.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-06 12:51:14 -07:00
Steve Howell c44500175d database: Remove short_name from UserProfile.
A few major themes here:

    - We remove short_name from UserProfile
      and add the appropriate migration.

    - We remove short_name from various
      cache-related lists of fields.

    - We allow import tools to continue to
      write short_name to their export files,
      and then we simply ignore the field
      at import time.

    - We change functions like do_create_user,
      create_user_profile, etc.

    - We keep short_name in the /json/bots
      API.  (It actually gets turned into
      an email.)

    - We don't modify our LDAP code much
      here.
2020-07-17 11:15:15 -07:00
Steve Howell 7c0fa3aefc populate_db: Add alert words for Zulip users. 2020-07-15 12:33:08 -07:00
Steve Howell 5c544da843 populate_db: Extract/fix mark_all_messages_as_read.
The prior version clobbered all flags, which means
we had unrealistic values for is_private.

Now we only touch the unread flag, which
also means when we go next to create alert words,
those will now work.
2020-07-15 12:33:08 -07:00
arpit551 e0c9b0fdff audit_log: Log RealmAuditLog when Stream is created.
Added new Event Type in AbstractRealmAuditLog STREAM_CREATED.
Since we finally create streams in create_stream_if_needed function
in zerver/lib/streams.py so logged realm_audit there.
Passed acting_user when create_stream_if_needed or ensure_stream
function  is called.
Added tests in test_audit_log.
2020-07-06 17:32:11 -07:00
arpit551 87aaa84b42 audit_log: Log acting_user in do_change_user_role. 2020-07-06 17:32:11 -07:00