Commit Graph

5024 Commits

Author SHA1 Message Date
sahil839 7cfbf660e6 actions: Remove deactivated stream from default stream groups.
Stream after deactivation should be removed from all the
default stream groups that it belongs to.

Fixes #14760.
2020-05-01 17:45:06 -07:00
jagansivam28 bbf5a5efed invitation: Make Member to see invitations sent by him/her.
Member of the org can able see list of invitations sent by him/her.
given permission for the member to revoke and resend the invitations
sent by him/her and added tests for test member can revoke and resend
the invitations only sent by him/her.

Fixes #14007.
2020-05-01 17:42:49 -07:00
Rohitt Vashishtha 7d3a31cd8b bugdown: Support hanging_lists preprocessor for indented lists.
Previously, hanging_lists preprocessor didn't consider anything
indented at 4 or above spaces to be a list. This meant that when
we had a list like:

1. 1
  2. 2
    3. 3
  2. 2a
1. 1a

We would insert a newline between 3. 3 and 2. 2a. This resulted
in the block processor breaeking down 1 list into 2 blocks, which
messed up the nesting and indentation for the second block.
2020-04-30 17:54:40 -07:00
Abhishek-Balaji 43e39718c4 user_name: Prevent users from setting name ending with |number.
We've had bugs in the past where users with a name in the format
"Alice|999" would confuse our markdown rendering or typeahead.  While
that's a fully solvable problem, there's no real use case for that, so
it's probably simpler to just prevent users from setting their name
that way.

Fixes #13923.
2020-04-30 15:59:12 -07:00
Wyatt Hoodes 5a58b7c549 data exports: Keep deleted export in UI table.
It makes sense to keep a deleted export in the table,
along with the time of deletion, for auditing reasons.
2020-04-30 13:00:59 -07:00
Wyatt Hoodes 82e7ad8e25 data exports: Handle pending and failed exports.
Prior to this change, there were reports of 500s in
production due to `export.extra_data` being a
Nonetype.  This was reproducible using the s3
backend in development when a row was created in
the `RealmAuditLog` table, but the export failed in
the `DeferredWorker`.  This left an entry lying
about that was never updated with an `extra_data`
field.

To fix this, we catch any exceptions in the
`DeferredWorker`, and then update `extra_data` to
encode the failure.  We also fix the fact that we
never updated the export UI table with pending exports.

These changes also negated the use for the somewhat
hacky `clear_success_banner` logic.
2020-04-30 13:00:59 -07:00
Aman Agrawal 5fa1dbf5b3 test_fixtures: Add DB function to write database digest.
This will give help up write new digest only if the db rebuild
succeeds. We were relying on the caller to
be successful in building db, this was hacky and unreliable.

We write new db digest once the caller succeeds, this ensures
that we write new digest after every successful attempt.

This fixes the anomality we were facing that Databases were rebuild
on the 2nd provision attempt with no changes to files or migrations.
This was happening because we didn't write a new digest for db
after the first provision (The case of DB didn't exist).

During the 1st provision, we check the template_status() of
Database both Dev and Test, but database_exists() of Databases
obviously returned false, and we rebuild the database,
but forgot to write_new_digest and hence the anomaly in the
second provision explained above.
2020-04-30 10:46:53 -07:00
Kartik Srivastava 7be52449a8 lib/bugdown/api_code_examples: Fix buggy rendering of print statement.
This fixes the buggy rendering of the print statement in the
python code examples.
2020-04-29 23:21:42 -07:00
Tim Abbott 8bba5dc6b6 test_fixtures: Include LOCAL_DATABASE_PASSWORD in digest.
This ensures that if one deletes `zproject/dev-secrets.conf`, we end
up rebuilding the databases from scratch (which, critically, will
ensure the password that gets setup matches what's in the current
version of the configuration file).

This should address a category of issue we've had where deleting
`zproject/dev-secrets.conf` would result in provision failing.
2020-04-29 22:57:37 -07:00
sahil839 3b99e3440e actions: Avoid unnecessary fake email address changes.
The logic in do_set_realm_property would previously "change" the email
addrssees of every user in the realm, even if they hadn't actually
changed.

We fix this by skipping the logic when it's unnecessary.
2020-04-29 16:50:08 -07:00
sahil839 ce6ea76110 actions: Use bulk_update when changing email_address_visibility.
bulk_update is used to update the email of user_profile objects in
database when email_address_visibility is changed.

This helps resolve the problem of timeout errors in realms with large
number of users due to large number of database queries run in a
loop.

Since bulk_update doesn't flush caches, we need our own bit of code to
do that.

Fixes a part of #14600.
2020-04-29 16:45:11 -07:00
Steve Howell 2c63130195 api: Always return zulip_version/zulip_feature_level.
We no longer make these conditional to simplify writing clients;
there's no cost to including them, and a real cost to not doing so.
2020-04-29 11:58:22 -07:00
Hashir Sarwar 3a110bd647 actions: Remove redundant 'url_embed_preview_enabled' check.
We add URLs to the `links_for_embed set`, only when
the `url_embed_preview_enabled` flag is turned on.
So, it is sufficient to check if `links_for_embed`
is not empty.
2020-04-28 22:41:00 -07:00
Mateusz Mandera f1ec02b40a auth: Add ExternalAuthResult to manage data in authentication flows.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).

The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.

This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.

Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
2020-04-28 22:19:02 -07:00
Vishnu KS 3f69500765 billing: Rename downgrade_realm_for_deactivation to downgrade_now. 2020-04-28 22:05:49 -07:00
Steve Howell cf78cb0d6e templates: Use `<hr>` and `<br>` consistently.
We now prevent these variations:

    * <hr/>
    * <hr />
    * <br/>
    * <br />

We could enforce similar consistency for other void
tags, if we wished, but these two are particularly
prevalent.
2020-04-28 17:05:48 -07:00
Puneeth Chaganti 88c297dffe tools: Allow specifying more options when capturing screenshot.
The name and directory into which the screenshot should be captured can be
specified now, apart from specifying the fixture file to use.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti 2c325e34ac integrations: Remove mutable default argument for config_options. 2020-04-28 13:00:09 -07:00
Puneeth Chaganti 39fe206289 integrations: Change legacy arg to a bool from Optional[bool]. 2020-04-28 13:00:09 -07:00
orientor 64c6bab276 openapi: Create markdown extension for rendering endpoint descriptions.
Add function in openapi.py to access endpoint descriptions written
in zulip.yaml. Use this function for creating a markdown extension
for rendering endpoint descriptions written in zulip.yaml.

We use this extension for a single endpoint to get test coverage.
2020-04-28 12:57:19 -07:00
Steve Howell 1aaef75bc5 test databases: Fix dev/test typo.
This line was changed to the wrong tool by
accident in 23f09fadfa.
2020-04-28 07:13:22 -04:00
Tim Abbott 03fecba917 alert_words: Fix cache flushing behavior and add tests.
The post_init cache-flushing behavior in the original alert words
migration was subtly wrong; while it may have passed tests, it didn't
have the right ordering for unlikely races.

We use post_save rather than post_init hooks precisely because they
ensure that we flush the cache after we know the database has been
updated and any future reads from the database will have the latest
state.
2020-04-27 11:48:32 -07:00
Tim Abbott 8e5b0351b3 alert_words: Fix case-sensitivity of alert words.
Previously, alert words were case-insensitive in practice, by which I
mean the Markdown logic had always been case-insensitive; but the data
model was not, so you could create "duplicate" alert words with the
same words in different cases.  We fix this inconsistency by making
the database model case-insensitive.

I'd prefer to be using the Postgres `citext` extension to have
postgres take care of case-insensitive logic for us, but that requires
installing a postgres extension as root on the postgres server, which
is a pain and perhaps not worth the effort to arrange given that we
can achieve our goals with transaction when adding alert words.

We take advantage of the migrate_alert_words migration we're already
doing for all users to effect this transition.

Fixes #12563.
2020-04-27 11:31:51 -07:00
Abhishek-Balaji 052368bd3e alert_words: Move alert_words from UserProfile to separate model.
Previously, alert words were a JSON list of strings stored in a
TextField on user_profile.  That hacky model reflected the fact that
they were an early prototype feature.

This commit migrates from that to a separate table, 'AlertWord'.  The
new AlertWord has user_profile, word, id and realm(denormalization so
we can provide a nice index for fetching all the alert words in a
realm).

This transition requires moving the logic for flushing the Alert Words
caches to their own independent feature.

Note that this commit should not be cherry-picked without the
following commit, which fixes case-sensitivity issues with Alert Words.
2020-04-27 11:29:50 -07:00
Abhishek-Balaji 818776faae alert bugdown: Change name of AlertWordsNotificationProcessor.
This is a precursor commit to change the name of
AlertWordNotificationProcessor to AlertWordsNotificationProcessor
to match the change from UserProfile.alert_words to Alertword.
2020-04-27 10:45:18 -07:00
Rohitt Vashishtha 087b54aa45 markdown: Keep only 'text' for a no-op pygments rendering.
Previously, we added support for 'none', 'plain' and 'noop' and a
function `lang = remap_language(lang)`. This also had the potential
to encourage adding more remappings- something that we deliberatly
want to keep to a minimum.

For context, Anders K doesn't want us to keep any remapping (only
keeping 'text' which is the default no-op lexer that pygments has)
and Tim wants to keep 'plain' and 'text'. We should only document
and advertise 'text'.
2020-04-27 10:40:29 -07:00
rebtung f7fbe3419f api: Improve consistency of reactions API.
Previously, the message and event APIs represented the user differently
for the same reaction data. To make this more consistent, I added a
user_id field to the reaction dict for both messages and events. I
updated the front end to use the user_id field rather than the user
dict. Lastly, I updated front end and back end tests that used user
info.

I primarily tested this by running my local Zulip build and
adding/removing reactions from messages.

Fixes #12049.
2020-04-26 10:35:55 -07:00
Puneeth Chaganti 2a65be2bf5 url preview: Use Chrome's user agent instead of a Zulip one.
Some sites don't render correctly unless you are one of the latest browsers.
YouTube Music, for instance, changes the page title to "Your browser is
deprecated, please upgrade.", which makes our URL previews look bad.
2020-04-26 10:16:43 -07:00
Rohitt Vashishtha ce6c57b714 markdown: Set default code block language for quotes and latex as well.
In the original implementation, we were checking for the default language
inside format_code, which resulted in the setting being ignored when set to
quote, math, tex or latex. We shift the validation to `check_for_new_fence`

We also update the tests to use a saner naming scheme for the variables.
2020-04-23 17:51:01 -07:00
Vishnu KS 22b5297e32 actions: Mark strings in send_welcome_bot_response for translation. 2020-04-23 17:19:27 -07:00
Vishnu KS 03286012d2 onboarding: Mark strings in send_initial_realm_messages for translation. 2020-04-23 17:18:39 -07:00
Vishnu KS 9a2ee29ac7 onboarding: Mark strings in send_initial_pms for translation. 2020-04-23 17:15:31 -07:00
sahil839 3a7de8ad3b models: Add has_permission as a generic function for different policies.
This commit removes can_create_streams and can_subscribe_other_users
to use has_permission as a generic function in UserProfile model for
these settings policies.

Relevant changes are made to events.py to avoid duplication at some
places.
2020-04-22 17:48:52 -07:00
Puneeth Chaganti 87648596ee integrations: Add script to create bot avatars for integrations.
These avatars can be directly used by bots used for documenting these
integrations. In future, these could be used by bots created from a web UI.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti 679f092a75 integrations: Extract code to get logo path for integrations. 2020-04-22 17:45:30 -07:00
Steve Howell 02252c255a db tools: Use common scheme for digests.
We have two different digest schemes to make
sure we keep the database up to date.  There
is the migration digest, which is NOT in the
scope of this commit, and which already
used the mechanism we use for other tools.

Here we are talking about the digest for
important files like `populate_db.py`.

Now our scheme is more consistent with how we
check file changes for other tools (as
well as the aformentioned migration files).

And we only write one hash file, instead of
seven.

And we only write the file when things have
actually changed.

And we are explicit about side effects.

Finally, we include a couple new bot settings
in the digest:

    INTERNAL_BOTS
    DISABLED_REALM_INTERNAL_BOTS

NOTE: This will require a one-time transition,
where we rebuild both databases (dev/test).
It takes a little over two minutes for me,
so it's not super painful.

I bump the provision version here, even
though you don't technically need it (since
the relevant tools are actually using the
digest files to determine if they need to
rebuild the database).  I figure it's just
good to explicitly make this commit trigger
a provision, and the user will then see
the one-time migration of the hash files
with a little bit less of a surprise.

And I do a major bump, not a minor bump,
because when we go in the reverse direction,
the old code will have to rebuild the
database due to the legacy hash files not
being around, so, again, I just prefer it
to be explicit.
2020-04-22 14:41:42 -07:00
Mateusz Mandera ec6022a186 logging_util: Handle record.status_code missing in skip_200_and_304.
Fixes #14595.
Invalid HTTP requests could end up in an unhandled exception in
skip_200_and_304 due the record not having the status_code attribute
set. With this change we'll avoid the exception

Example:
curl  -X POST -H 'Transfer-Encoding : chunked' --data-binary 'a' 'http://zulipdev.com:9991/json/messages/57'

2020-04-21 10:56:22.007 WARN [django.server] "POST /json/messages/57 HTTP/1.1" 405 95
2020-04-21 10:56:22.007 INFO [django.server] code 400, message Bad request syntax ('a')
2020-04-21 10:56:22.008 WARN [django.server] "a" 400 -
2020-04-22 11:26:13 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Steve Howell 239474124e test-backend: Remove generate_fixtures option.
We remove the `generate_fixtures` option here mostly
for simplicity, but in particular to facilitate
an upcoming commit to simplify the job of
`generate-fixtures` (and remove its `--force` option).

The command line option here for `test-backend`
was really calling `generate_fixtures --force`,
which we're about to rename `tools/rebuild-test-database`.

The `test-backend` tools is already smart about catching
up on migrations, so we generally don't need to tell it
to repair the database.

And if the database does get corrupt, you can just do
it directly with `tools/rebuild-test-database`.

This eliminates the `use_force` flag in
`update_test_databases_if_required`, which was easy
to confuse with `rebuild_test_database`.

The other caller wasn't using `use_force`.
2020-04-22 09:24:42 -07:00
Steve Howell 23f09fadfa refactor: Use run in update_test_databases_if_required.
Use `run` to run the tools, and take advantage
that `rebuild-dev-database` is really the same
as `generate-fixtures --force`.
2020-04-22 09:24:42 -07:00
Steve Howell b10be1f8b7 refactor: Early-exit in update_test_databases_if_required.
Just make each conditional run what it needs to run.  The
simplicity that this provides will be more apparent
soon.
2020-04-22 09:24:42 -07:00
Steve Howell 67d0349239 refactor: Use precise names for migration helpers.
Somewhat confusingly, we have two types of different
digests related to databases.  The migration digests
are pragmatic, since changes to migrations are a bit
more frequent for certain use cases and don't
necessitate a complete rebuild of the database.

Anyway, these are just more specific names.
2020-04-22 09:24:42 -07:00
Anders Kaseorg 088f7ee5d6 python: Convert type checks to isinstance checks.
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532.  In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Kartik Srivastava 18b577f600 lib/bugdown/api_code_examples: Refactor extract_python_code_example.
This refactors `extract_python_code_example` to accept an
`example_regex` parameter. It can now be used to extract code examples
from javascript_examples.py.
2020-04-21 17:31:07 -07:00
Tim Abbott 703fae8980 send_custom_email: Use a special .gitignored directory.
Previously, the send_custom_email code path leaked files in paths that
were not `.gitignored`, under templates/zerver/emails.

This became problematic when we added automated tests for this code
path, as it meant we leaked these files every time `test-backend` ran.

Fix this by ensuring all the files we generate are in this special
subdirectory.
2020-04-21 16:50:11 -07:00
Hashir Sarwar e3b90a5ec8 api: Add a monotonic integer "feature level" for non-webapp clients.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that

* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;

* is reliable, unlike e.g. looking at the number of
commits since a release;

* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.

Tweaked by tabbott to extend the documentation.

Closes #14618.
2020-04-21 13:37:57 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00