Commit Graph

2671 Commits

Author SHA1 Message Date
Hashir Sarwar 55a8e7dff2 settings: Offer hiding presence info from other users.
For privacy-minded folks who don't want to leak the
information of whether they're online, this adds an
option to disable sending presence updates to other
users.

The new settings lies in the "Other notification
settings" section of the "Notification settings"
page, under a "Presence" subheading.

Closes #14798.
2020-05-02 15:09:05 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
jagansivam28 bbf5a5efed invitation: Make Member to see invitations sent by him/her.
Member of the org can able see list of invitations sent by him/her.
given permission for the member to revoke and resend the invitations
sent by him/her and added tests for test member can revoke and resend
the invitations only sent by him/her.

Fixes #14007.
2020-05-01 17:42:49 -07:00
Anders Kaseorg a552c2e5f9 auth: Use the clipboard instead of zulip:// for desktop auth flow.
This does not rely on the desktop app being able to register for the
zulip:// scheme (which is problematic with, for example, the AppImage
format).

It also is a better interface for managing changes to the system,
since the implementation exists almost entirely in the server/webapp
project.

This provides a smoother user experience, where the user doesn't need
to do the paste step, when combined with
https://github.com/zulip/zulip-desktop/pull/943.

Fixes #13613.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-04-30 16:45:00 -07:00
Wyatt Hoodes 82e7ad8e25 data exports: Handle pending and failed exports.
Prior to this change, there were reports of 500s in
production due to `export.extra_data` being a
Nonetype.  This was reproducible using the s3
backend in development when a row was created in
the `RealmAuditLog` table, but the export failed in
the `DeferredWorker`.  This left an entry lying
about that was never updated with an `extra_data`
field.

To fix this, we catch any exceptions in the
`DeferredWorker`, and then update `extra_data` to
encode the failure.  We also fix the fact that we
never updated the export UI table with pending exports.

These changes also negated the use for the somewhat
hacky `clear_success_banner` logic.
2020-04-30 13:00:59 -07:00
Hashir Sarwar 3a110bd647 actions: Remove redundant 'url_embed_preview_enabled' check.
We add URLs to the `links_for_embed set`, only when
the `url_embed_preview_enabled` flag is turned on.
So, it is sufficient to check if `links_for_embed`
is not empty.
2020-04-28 22:41:00 -07:00
Mateusz Mandera f1ec02b40a auth: Add ExternalAuthResult to manage data in authentication flows.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).

The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.

This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.

Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
2020-04-28 22:19:02 -07:00
akashaviator 28d58c848d api: Guess emoji_type when adding a reaction.
This changes add_reaction in zerver.views.reactions to allow
calling POST ../messages/{message_id}/reactions api endpoint with
emoji_name only, even in the case of a custom emoji.
2020-04-28 20:49:39 -07:00
Puneeth Chaganti f280b64caa integrations: Remove hasattr check for stream_name on Integrations.
`stream_name` is always set on `Integration`. If it is not explicitly
specified, it is set to the name of the integration.
2020-04-28 13:00:09 -07:00
Tim Abbott 976e554799 Revert "home: Don't use pointer to compute furthest_read_time."
This reverts commit 8f32db81a1.

This change unfortunately requires an index that we don't have, and
thus is incredibly expensive.  We'll need to do a thoughtful reworking
before we can integrate it again.
2020-04-27 22:29:39 -07:00
Puneeth Chaganti 8f32db81a1 home: Don't use pointer to compute furthest_read_time.
When a user is reading messages only in stream or topic narrows, the pointer
can be left far behind. Using this to compute the furthest_read_time causes
the banckruptcy banner to be shown even when a user has been actively
reading messages. This commit switches to using the sent time on the last
message that the user has read to compute the furthest read time.
2020-04-26 22:41:52 -07:00
Tim Abbott d8def0de55 pointer: Remove useless last_pointer_updater field.
This hasn't done anything in at least 5 years.
2020-04-26 22:39:24 -07:00
Tim Abbott 731cd79436 messages: Remove first_unread pointer hack.
This hack was important when only the mobile apps (and not the webapp)
were using the unread_msgs data structure and the first_unread
infrastructure.  Now that the webapp is using those things, there
aren't leaked ancient unread messages that aren't accessible on the
webapp, so any few users still in this situation can get out of it by
just reading the problematic messages.
2020-04-26 22:37:41 -07:00
Dinesh 2735860f01 browser_support: Drop support for Internet Explorer.
Internet Explorer does not support `position: sticky` which improves
floating recipient bar behavior during scrolling which is one of the
issues blocking PR #9910.
IE also does not support some features that modern browsers support
hence may not super well.
This commit adds an error page that'll be displayed when a user logs
in from Internet Explorer. Also, a test is added.
2020-04-23 17:50:28 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Kartik Srivastava b29ccdf51c
api: Refactor get_members_backend to return a single bot's data.
This makes `get_members_backend` in zerver/views/users.py to
return a single bot's data too.
2020-04-21 13:45:58 -07:00
Hashir Sarwar e3b90a5ec8 api: Add a monotonic integer "feature level" for non-webapp clients.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that

* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;

* is reliable, unlike e.g. looking at the number of
commits since a release;

* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.

Tweaked by tabbott to extend the documentation.

Closes #14618.
2020-04-21 13:37:57 -07:00
Mateusz Mandera 4018dcb8e7 upload: Include filename at the end of temporary access URLs. 2020-04-20 10:25:48 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Udit107710 db30cf470c refactor: Making email_mirror independent of actions.
Moved truncate_body, truncate_content and truncate_topic
to message.py.
2020-04-18 16:58:29 -07:00
Divyanshu Agrawal 8a64ed6e20 alert words: Limit alert word length to 100.
We do limit the length to 100 in the frontend, but no such check
exists in the backend.

Check that a new alert word has a maximum length of 100 in the
alert_words endpoint.
2020-04-17 11:32:16 -07:00
Tim Abbott 0ccc0f02ce upload: Support requesting a temporary unauthenticated URL.
This is be useful for the mobile and desktop apps to hand an uploaded
file off to the system browser so that it can render PDFs (Etc.).

The S3 backend implementation is simple; for the local upload backend,
we use Django's signing feature to simulate the same sort of 60-second
lifetime token.

Co-Author-By: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-04-17 09:08:10 -07:00
Tim Abbott d30c8d4a66 terms: Fix email address visibility displayed when accepting ToS.
Apparently, we never tested this unusual flow (primarily used after
Slack import) with an organization with EMAIL_ADDRESS_VISIBILITY_ADMINS.
2020-04-16 11:44:51 -07:00
Kartik Srivastava a060691389 default streams: Change add api to use stream_id.
This refactors add_default_stream in zerver/views/streams.py to
take in stream_id as parameter instead of stream_name.

Minor changes have been made to test_subs.py and settings_streams.js
accordingly.
2020-04-15 15:47:04 -07:00
Puneeth Chaganti 67822a0544 messages: Fix 500 on queries with empty string as search operand.
The error is PGroonga specific since `pgroonga_query_extract_keywords` does
not handle empty string inputs correctly. This commit prevents search
narrows from having empty operands.

Closes #14405
2020-04-12 11:04:09 -07:00
Tim Abbott 59937ec9fa scripts: Rename inline-email-css to inline_email_css.py.
This is a preparatory step for making it a module that we can import
and call from other code.
2020-04-10 15:29:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Rohitt Vashishtha f9caf522f0 markdown: Allow setting a default language for code blocks.
This adds a new realm setting: default_code_block_language.

This PR also adds a new widget to specify a language, which
behaves somewhat differently from other widgets of the same
kind; instead of exposing methods to the whole module, we
just create a single IIFE that handles all the interactions
with the DOM for the widget.

We also move the code for remapping languages to format_code
function since we want to preserve the original language to
decide if we override it using default_code_clock_language.

Fixes #14404.
2020-04-09 16:02:02 -07:00
Vishnu KS 449f7e2d4b team: Generate team page data using cron job.
This eliminates the contributors data as a possible source of
flakiness when installing Zulip from Git.

Fixes #14351.
2020-04-08 12:52:31 -07:00
Mateusz Mandera 9911c6a0f0 rate_limiter: Put secs_to_freedom as message when raising RateLimited.
That's the value that matters to the code that catches the exception,
and this change allows simplifying the plumbing somewhat, and gets rid
of the get_rate_limit_result_from_request function.
2020-04-08 10:29:18 -07:00
Tim Abbott 843345dfee message_edit: Add backend for moving a topic to another stream.
This commit reuses the existing infrastructure for moving a topic
within a stream to add support for moving topics from one stream to
another.

Split from the original full-feature commit so that we can merge just
the backend, which is finished, at this time.

This is a large part of #6427.

The feature is incomplete, in that we don't have real-time update of
the frontend to handle the event, documentation, etc., but this commit
is a good mergable checkpoint that we can do further work on top of.
We also still ideally would have a test_events test for the backend,
but I'm willing to leave that for follow-up work.

This appears to have switched to tabbott as the author during commit
squashing sometime ago, but this commit is certainly:

Co-Authored-By: Wbert Adrián Castro Vera <wbertc@gmail.com>
2020-04-07 14:19:19 -07:00
Mateusz Mandera 8686bbf637 auth: Show a user-facing page when wrong token given to /subdomain/.
This used to show a blank page. Considering that the links remain valid
only for 15 seconds it's important to show something more informative to
the user.
2020-04-05 12:29:09 -07:00
Tim Abbott e27cafd785 settings: Don't use nice REQ syntax for language_codes.
This breaks provisioning because running this as import time would
require language_name_map.json to be generated by `manage.py
compilemessages` before we can run any management commands :(.

We could potentially fix this in the future by changing the generate
language files to be things we commit to the project.
2020-04-03 16:00:47 -07:00
sahil839 b66a67272f models: Reuse constants for common realm permissions policies.
This is a prep commit for making use of same choices for
create_stream_policy and invite_to_stream_policy as both fields
have same set of choices.

This will be useful as we add other fields using these same types.
2020-04-03 15:17:51 -07:00
sahil839 386215614c models: Replace WAITING_PERIOD with FULL_MEMBERS in choice variables.
This commit replaces the WAITING _PERIOD with FULL_MEMBERS from
create_stream_policy and invite_to_stream_policy choices to
achieve consistency and making the variables more descriptive.
2020-04-03 15:14:13 -07:00
Tim Abbott a745e533fe settings: Use cleaner validators for display settings.
This simplifies the update_display_settings endpoint to use REQ for
validation, rather than custom if/else statements.

The test changes just take advantage of the now more consistent
syntax.
2020-04-03 15:09:14 -07:00
Tim Abbott 17723b1617 version: Move minimum desktop version configuration to version.py.
This makes it relatively easy for a system administrator to
temporarily override these values after a desktop app security
release that they want to ensure all of their users take.

We're not putting this in settings, since we don't want to encourage
accidental long-term overrides of these important-to-security values.
2020-04-01 13:23:08 -07:00
Steve Howell 2fe21de733 dev login: Sort users. 2020-03-30 10:16:27 -04:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Tim Abbott 737a36a9f8 portico: Use a backend variable to determine desktop version.
This makes it possible to change this parameter without rebuilding all
the server's static assets.
2020-03-27 01:37:56 -07:00
Anders Kaseorg 56aadf6503 tests: Fix CI apocalypse perpetrated by previous commit.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:25:25 -07:00
Steve Howell 697b00dd6e default streams: Change remove api to use stream_id.
This is a full-stack change:

    - server
    - JS code
    - templates

It's all pretty simple--just use stream_id instead
of stream_name.

I am 99% sure we don't document this API nor use it
in mobile, so it should be a safe change.
2020-03-25 17:11:25 -07:00
Graham Bleaney 2fe9d85a5f redirects: Refactor redirect code to use central helper function.
This commit introduces two new functions in 'url_encoding.py' which
centralize two common patterns for constructing redirect URLs. It
also migrates the files using those patterns to use the new
functions.
2020-03-25 16:39:17 -07:00
Graham Bleaney 3e602a9bd4 integrations: Harden development views against path traversal.
This commit hardens the code in the developer integrations views by
ensuring that file paths can only be constructed with valid
integration names.
2020-03-25 16:39:17 -07:00
Tim Abbott d9bb6d0081 compatibility: Add more strict desktop app blocking.
This allows us to block use of the desktop app with insecure versions
(we simply fail to load the Zulip webapp at all, instead rendering an
error page).

For now we block only versions that are known to be both insecure and
not auto-updating, but we can easily adjust these parameters in the
future.
2020-03-24 20:33:11 -07:00
Tim Abbott 85c9ffd91c message: Validate propagate_mode parameters.
This improves the error handling for invalid values of the
propagate_mode parameter to our message editing endpoints.
Previously, invalid values would just work like change_one rather than
doing nothing.
2020-03-24 12:36:45 -07:00
Dinesh 5888d7c0f5 auth: Change how config error URLs are configured.
URLs for config errors were configured seperately for each error
which is better handled by having error name as argument in URL.
A new view `config_error_view` is added containing context for
each error that returns `config_error` page with the relevant
context.
Also fixed tests and some views in `auth.py` to be consistent with
changes.
2020-03-22 17:15:18 -07:00
Steve Howell a041d9e4aa minor: Clean up lstrip() for help article titles.
Saying `foo.lstrip('# ')` does more than just remove
a '# ' prefix.  It removes any combination of '#' and
spaces.

We now make the intention slightly more clear.

We would strip these as you'd expect:

    # foo
    ## foo
    ### foo

but for this we now only strip the first "#":

    # # # # # foo
2020-03-22 11:32:29 -07:00
Steve Howell edf1b1e5e8 minor: Fix buggy lstrip() call in integrations dev panel.
Thanks to @minusworld for catching this--see #14264, which
points out that lstrip() doesn't do what your intuition
might tell you it does.

Now we properly remove the "HTTP_" prefix.

It's not clear to me why we need these prefixes for Django
purposes in the fixtures, but I didn't want to go down
the rabbit hole of fixing those.

To test:

    got to http://YOUR-DEV_SERVER/devtools/integrations/
    select "bitbucket3" for the integration.
    select "diagnostics_ping.json" for the fixture.
    see "X_EVENT_KEY" in "Custom HTTP Headers"

Fixes #14264
2020-03-22 11:32:29 -07:00
Steve Howell d71111f3dc presence api: Use email to look up presence.
We don't want to use delivery_email to look up
presence on email-restricted realms.
2020-03-19 14:32:18 -07:00
Mateusz Mandera f5e95c4fc1 requirements: Bump python-social-auth version.
We had a bunch of ugly hacks to monkey patch things due to upstream
being temporarily unmaintained and not merging PRs. Now the project is
active again and the fixes have been merged and included in the latest
version - so we clean up all that code.
2020-03-18 12:14:31 -07:00
Tim Abbott b064559652 zephyr: Add strict assertion about username format.
This ensures that even if it were possible to create an MIT Kerberos
account with a malicious username and/or hack webathena to pretend
that's the case, one couldn't do anything malicious.

This security improvement only impacts a single installation of Zulip
where Zephyr mirroring is in use that has already had the fix applied,
so there's no reason to do a security notice for it.

Found by Graham Bleaney using pysa.
2020-03-17 05:37:25 -07:00
Mateusz Mandera 2000608a9e report_error: Fix inaccurate docstring.
do_report_error isn't actually below.
2020-03-09 13:54:58 -07:00
Mateusz Mandera 0255ca9b6a middleware: Log user.id/realm.string_id instead of _email. 2020-03-09 13:54:58 -07:00
akashaviator 9c63976da5 api: Refactor get_members_backend in zerver/views/users.py.
This refactors get_members_backend to return user data of a single
user in the form of a dictionary (earlier being a list with a single
dictionary).

This also refactors it to return the data with an appropriate key
(inside a dictionary), "user" or "members", according to the type of
data being returned.

Tweaked by tabbott to use somewhat less opaque code and simple OpenAPI
descriptions.
2020-03-08 18:43:30 -07:00
Tim Abbott 2c75b39078 templates: Delete show_debug feature.
As far as I know, this hasn't been used in at least 5 years, and I'm
not sure there's a real use case for it with the current app.
2020-03-08 18:34:59 -07:00
Tim Abbott 53cc00c21c messages: Ban the sender property when not mirroring.
The "sender" property in `send_message_backend` is meant to only do
something when doing Zephyr mirroring (or similar).  We should help
clients behave correctly by banning this property in requests that are
not specifically requesting mirroring behavior.

This commit requires changes to a number of tests that incorrectly
passed this parameter or didn't use the right setup for mirroring.
2020-03-08 14:09:32 -07:00
Tim Abbott 9230213bde settings: Add EMAIL_ADDRESS_VISIBILITY_NOBODY.
This extends our email address visibility settings to deny access to
user email addresses even to organization administrators.

At the moment, they can of course change the setting (which leaves an
audit trail), but in the future only organization owners will be able
to change that setting.

While we're at this, we rewrite the settings_data.js test to cover all
the cases in a more consistent way.

Fixes #14111.
2020-03-06 16:34:08 -08:00
Steve Howell f2b8eef21a refactor: Avoid hacky use of ValidationError.code.
We were using `code` to pass around messages.

The `code` field is designed to be a code, not
a human-readable message.

It's possible that we don't actually need two
flavors of messages for these type of validations,
but I didn't want to change that yet.

We **definitely** don't need to put two types of
message in the exception, so I fix that.  Instead,
I just have the caller ask what level of detail
it needs.

I added a non-verbose message for the case of
system bots.

I removed the non-translated version of the message
for deactivated accounts, which didn't have test
coverage and is slightly more prone to leaking
email info that we don't want to leak.
2020-03-06 11:53:22 -08:00
Steve Howell 62fb3ad801 refactor: Move validate_email_not_already_in_realm.
We move this to email_validation.py.
2020-03-06 11:53:22 -08:00
Steve Howell ad85e286de user settings: Inline call to validate_email.
We are trying to elminate the version of
`validate_email` that lives in `actions.py`.

Inlining it barely increases the code size, and
it removes some noise related the three-item
tuple that `check_incoming_email` returns.
2020-03-06 11:53:22 -08:00
Steve Howell 4f5b07a7e6 refactor: Extract zerver/lib/email_validation.py. 2020-03-06 11:53:22 -08:00
Steve Howell 57f1aa722c refactor: Rename validate_email_for_realm.
Now called:

    validate_email_not_already_in_realm

We have a separate validation function that
makes sure that the email fits into a realm's
domain scheme, and we want to avoid naming
confusion here.
2020-03-06 11:53:22 -08:00
Mateusz Mandera 6e4dcc714e auth: Fix bug in backend configuration checks in start_social_login.
We shouldn't check if *every* backend in the backends list is
configured, but only the one the user is attempting to log in with.
2020-03-01 22:30:38 -08:00
Mateusz Mandera 2d55c7e0d5 home: Don't assume user agent header is set for insecure_desktop_app.
The header may not be set - this leads to CI failures on 2.1.x branch,
but in any case is a real bug.
2020-02-28 16:51:23 -08:00
Tim Abbott d79a7a8c35 panels: Show a banner for users with legacy desktop apps.
Users who are using ZulipDesktop or haven't managed to auto-update to
ZulipElectron should be strongly encouraged to upgrade.

We'll likely want to move to something even stricter that blocks
loading the app at all, but this is a good start.
2020-02-28 01:54:46 -08:00
Mateusz Mandera efb3065158 social_auth: Take user to find_account if invalid subdomain is given.
This allows to also clean up some code that's not really useful.
2020-02-27 17:27:55 -08:00
Anders Kaseorg 1cdab5ae61 emoji: Resolve emoji sprite sheets and stylesheets through Webpack.
This gives them cache-compatible URLs, and also avoids some extra
copies of the sprite sheet images.

Comments on the Octopus emoji added by tabbott.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:43:46 -08:00
Steve Howell 29dd7d2267 minor: Fix comment in send_message_backend.
The `send_message_backend` function no longer
calls `recipient_for_emails`.
2020-02-24 15:32:29 -08:00
Mateusz Mandera 2343f80d72 auth: Remove redundant get_subdomain(request) call in remote_user_sso.
subdomain = get_subdomain(request) is already called earlier in the
function.
2020-02-24 12:39:48 -08:00
Mateusz Mandera f2903e9c70 auth: Refactor - convert remote_username to email in remote_user_sso.
So far the conversion was in a very random place -
register_remote_user(). All other codepaths that use
login_or_register_remote_user() call it with the user's email address.
Making remote_user_sso convert remote_username to the email address
before calling login_or_register_remote_user makes this usage consistent
across the board.
2020-02-24 12:39:48 -08:00
Mateusz Mandera 98ae2fb940 auth: Remove redundant realm argument to finish_desktop_flow.
finish_desktop_flow is called with the assumption that the request
successfully proved control over the user_profile and generates a
special link to log into the user_profile account. There's no reason to
pass the realm param, as user_profile.realm can be assumed.
2020-02-24 12:39:48 -08:00
Dinesh 1308544a70 auth: Remove `if` blocks to redirect to config error page.
In `auth.py` there are three `if` blocks for different backends
to redirect to config error page with similar code. It is better
handled with common code using `get_attr()` function on
constructed setting names.
2020-02-24 12:19:45 -08:00
vsvipul 020a263a67 auth: Create a new page hop for desktop auth.
Create a new page for desktop auth flow, in which
users can select one from going to the app or
continue the flow in the browser.

Co-authored-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-02-20 11:59:55 -08:00
Vishnu KS e1a7716578 emails: Translate from_name of account security emails. 2020-02-18 17:45:33 -08:00
Mateusz Mandera 6a0b68bc7f models: Delete get_stream_recipient function and its uses.
With recipient being now a Stream field, there's no more use for
this helper function.
2020-02-18 10:49:14 -08:00
rht 41e3db81be dependencies: Upgrade to Django 2.2.10.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.

Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:

* urls: django.urls.resolvers.RegexURLPattern has been replaced by
  django.urls.resolvers.URLPattern; affects OpenAPI code and related
  features which re-parse Django's internals.
  https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
  ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
  https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
  https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
  the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.

There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.

Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.

Fixes #10835.
2020-02-13 16:27:26 -08:00
Mateusz Mandera bde495db87 registration: Add support for mobile and desktop flows.
This makes it possible to create a Zulip account from the mobile or
desktop apps and have the end result be that the user is logged in on
their mobile device.

We may need small changes in the desktop and/or mobile apps to support
this.

Closes #10859.
2020-02-12 11:22:16 -08:00
Mateusz Mandera 2475adbf8a messages_for_topic: Use stream.recipient_id for more efficient query. 2020-02-11 17:39:43 -08:00
Dinesh 4304d5f8db auth: Add support for GitLab authentication.
With some tweaks by tabbott to the documentation and comments.

Fixes #13694.
2020-02-11 13:54:17 -08:00
Tim Abbott fcac3a4342 recipients: Rename extract_recipients to extract_private_recipients.
Recent changes mean this function is now only used for private
messages.
2020-02-11 12:28:14 -08:00
Steve Howell 1b6578cafd messages: Fix bug with commas in stream names.
We now validate streams with a separate
function from PM recipients.

It's confusing enough all the ways you can
encode a stream or encode the PM recipients,
but trying to do it all in one function was
hard to reason about and led to at least one
bug.

In particular, there was a bug where streams
with commas in them would get split.  Now
we just don't ever split on commas inside
of `extract_stream_indicator`.

Fixes #13836
2020-02-11 12:20:54 -08:00
akashaviator 1ae5964ab8 api: Add an api endpoint for GET /users/{id}
This adds a new API endpoint for querying basic data on a single other
user in the organization, reusing the existing infrastructure (and
view function!) for getting data on all users in an organization.

Fixes #12277.
2020-02-07 10:36:31 -08:00
Tim Abbott 79e5dd1374 users: Rename get_raw_user_data user parameter to acting_user.
This is for improved clarity as we extend this function to take
multiple user objects.
2020-02-07 10:36:31 -08:00
Steve Howell bf3baa14ac presence: Rename get_status_dict_by_user(). 2020-02-06 17:16:22 -08:00
Steve Howell 36fba1076f presence: Move get_status_dict_by_user. 2020-02-06 17:16:22 -08:00
Steve Howell 703338dfa3 presence: Extract lib/presence.py.
This will make more sense when we pull some
code out of the model.
2020-02-06 17:16:22 -08:00
Steve Howell a5093be867 presence: Rename get_status_list.
The word "status" is vague, and this isn't
actually returning a list, so we now name it
get_presence_response.

I originally was gonna rename this to
get_presence_dict, but there's a function
called get_status_dict that returns a subset
of the response, so I think it's a bit more
clear that this is the bigger dict that
actually gets sent back.
2020-02-06 17:16:22 -08:00
Steve Howell 8a1fb2dcd6 presence: Calculate server_timestamp slightly earlier.
We want to err on the side of server_timestamp being
old, since we may eventually use this to make responses
just include incremental changes, and we don't want a
time window (however small) when we miss presence rows.
The clients will be able to deal with duplicate data
to the extent that the time windows are overlapping.

Also, extracting the other local var here
(for `presences`) will set up a subsequent commit
where we re-format the data for clients with
slim_presence=True.
2020-02-06 17:16:22 -08:00
Tim Abbott b5cc1a3afe home: Finish removing assumption that user_profile isn't None. 2020-02-05 16:16:33 -08:00
Tim Abbott be02604fe5 home: Refactor logic for several more settings.
The main functional change is there's now a code path for
`user_profile is None`.
2020-02-05 16:13:29 -08:00
Tim Abbott 5f8af84ad1 home: Refactor emojiset and billing logic for None UserProfile. 2020-02-05 16:09:03 -08:00
Tim Abbott fc4400cfa7 home: Remove unused avatar_url context parameter. 2020-02-05 16:05:02 -08:00
Tim Abbott 7044aa8e5d home: Extract updates to last_reminder to a function. 2020-02-05 16:05:02 -08:00
Tim Abbott 85690ce649 home: Extract handling of narrow windows to a function. 2020-02-05 16:05:02 -08:00
Tim Abbott 42d529e749 home: Extract terms of service check to a function. 2020-02-05 16:05:01 -08:00
Anders Kaseorg 4d49a20430 requirements: Upgrade django-sendfile2 from 0.4.3 to 0.5.1.
The module was renamed from sendfile to django_sendfile.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Mateusz Mandera 0e7c97378e is_safe_url: Use allowed_hosts instead of depreciated host argument.
Judging by comparing django 1.11 with django 2.2 code of this function,
this shouldn't change any behavior.
2020-02-04 12:46:53 -08:00
Steve Howell 9847d4d9a3 refactor: Use user_id in get_status_dict_by_user.
This avoids a needless user lookup in apply_event.
2020-02-04 12:30:36 -08:00
Steve Howell bf9144ff69 presence: Add slim_presence flag.
This flag affects page_params and the
payload you get back from POSTs to this
url:

    users/me/presence

The flag does not yet affect the
presence events that get sent to a
client.
2020-02-04 12:30:34 -08:00
Ryan Rehman 3dc7d60ffe muting: Record DateTime when a Topic is muted.
This includes the necessary migration to add
the date_muted field to the MutedTopic class
and populates it with a hard coded value.
2020-02-02 20:49:53 -08:00
Mateusz Mandera b685d506ad auth: Make login_context available to TwoFactorLoginView. 2020-02-02 20:22:49 -08:00
Mateusz Mandera 06198af5b9 auth: Handle rate limiting in OurAuthenticationForm and user_settings.
These parts of the code should catch the RateLimited exception and
generate their own, apprioprate user-facing error message.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 677764d9ca auth: Pass request kwarg in authenticate() calls with username+password.
These authenticate() calls use either Email or LDAP backends, which will
be rate limited and will need access to the request object.
2020-02-02 19:15:00 -08:00
Mateusz Mandera d5786ee67a auth: Ensure only one of mobile and desktop otps in validate_otp_params.
validate_otp_params needs to be moved to backends.py, because as of this
commit it'll be used both there and in views.auth - and import from
views.auth to backends.py causes circular import issue.
2020-02-02 19:14:40 -08:00
Mateusz Mandera f6301bf065 auth: Extract validate_otp_params function. 2020-02-02 19:14:40 -08:00
Mateusz Mandera eea68ce92d auth: Support desktop_flow_otp with remote_user_sso. 2020-02-02 19:14:40 -08:00
akashaviator f8bcadfc63 refactor: Combine import statements from zerver.lib.users
This combines two separate import statements from zerver.lib.users
,in zerver/views/users.py, into one.
2020-02-02 18:55:56 -08:00
akashaviator 97235725ec refactor: Make get_raw_user_data get imported from zerver.lib.users
This makes get_raw_user_data, which was being imported indirectly
from zerver.lib.events inside zerver/views/users.py, get imported
from zerver.lib.users where it actually is.
2020-02-02 18:55:56 -08:00
Mateusz Mandera 14cea57e60 auth: Replace deprecated password_reset. 2020-02-02 17:03:10 -08:00
Mateusz Mandera 3fec19d555 auth: Replace deprecated django.contrib.auth.views.login. 2020-02-02 17:03:10 -08:00
Ray Kraesig d7b900ca52 register: Ensure future client_capabilities fields are optional.
The `notification_settings_null` field of the `client_capabilities`
parameter is, apparently unintentionally, required.

This is mostly harmless. However, if any _future_ fields are made
required, all existing clients using this parameter will break, and it
will be needlessly difficult for new clients to specify new
capabilities in a backwards-compatible way.

Attempt to stave that possibility off with warnings.

(No functional changes.)
2020-01-31 15:59:23 -08:00
Mateusz Mandera c618f0770e social_auth: Clear session fields leftover from previous auth attempts.
Fixes #13560.
2020-01-30 14:45:12 -08:00
Tim Abbott b25fea24e7 messages: Simplify API for use_first_unread_anchor.
Now that we have the type situation of having anchor support passing a
string, this is a much more natural way to implement
use_first_unread_anchor.

We still support the old interface to avoid breaking compatibility
with legacy versions of the mobile apps.
2020-01-29 12:17:03 -08:00
Tim Abbott 05108760f6 narrow: Add support for passing oldest/newest for anchor.
A wart that has long been present inin Zulip's get_messages API is how
to request "the latest messages" in the API.  Previously, the
recommendation was basically to pass anchor=10000000000000000 (for an
appropriately huge number). An accident of the server's implementation
meant that specific number of 0s was actually important to avoid a
buggy (or at least wasteful) value of found_newest=False if the query
had specified num_after=0 (since we didn't check).

This was the cause of the mobile issue
https://github.com/zulip/zulip-mobile/issues/3654.

The solution is to allow passing a special value of anchor='newest',
basically a special string-type value that the server can interpret as
meaning the user precisely just wants the most recent messages.  We
also add an analogous anchor='oldest' or similar to avoid folks
needing to write a somewhat ugly anchor=0 for fetching the very first
messages.

We may want to also replace the use_first_unread_anchor argument to be
a "first_unread" value for the anchor parameter.

While it's not always ideal to make a value have a variable type like
this, in this case it seems like a really clean way to express the
idea of what the user is asking for in the API.
2020-01-29 12:14:06 -08:00
Greg Price a5aa541999 portico: Provide isolated single-page versions of /terms and /privacy .
The `isolated_page` context flag we rely on was added in the
parent commit.
2020-01-29 11:54:20 -08:00
Tim Abbott bcbc8f2bd5 portico: Move portico view code to its own file.
This improves the readability of the codebase.
2020-01-29 11:54:20 -08:00
Tim Abbott e2810d7549 narrow: Fix server handling of other larger anchor values.
This fixes a bug where that clients using the legacy approach of a
"very large anchor" value with the intent to only get the most recent
messages would only get found_newest=True if they used the specific
value LARGER_THAN_MAX_MESSAGE_ID.  Now any value at least that large
will work.

In upcoming commits, we plan to replace this with passing the string
"last", but it seems worth removing the buggy "special value" behavior
while we're touching this code.
2020-01-29 11:24:58 -08:00
Tim Abbott a3f08f01ec django: Use request.user.is_authenticated consistently.
In Django 2.0, request.user.is_authenticated stops supporting
`.is_authenticated()` and becomes just a property.  In 1.11, it's a
CallableProperty (i.e. can be used either way), and we already use it
as a property in several other places, so we should just switch to
using it consistently now to get it off of our Django 2.x migration
checklist.
2020-01-27 17:50:18 -08:00
Tim Abbott 1655aabe65 auth: Simplify logic around streams_to_subscribe.
This avoids some typing issues we'll have to resolve before upgrading
to Django 2.
2020-01-27 17:20:17 -08:00
Hashir Sarwar 13b3eb24b0 invitations: Avoid adding to notifications stream unconditionally.
Adding invited users to the notifications stream unconditionally isn't
a correct behaviour for guest users, where the previous behavior of
including the notifications stream no longer makes sense. Therefore,
while inviting a new user, the notifications stream is listed along
with other streams with a message "recieves notifications for new
streams" in order to distinguish it from other streams.

Fixes #13645.
2020-01-27 15:36:59 -08:00
Mateusz Mandera 92c16996fc redis_utils: Require key_format argument in get_dict_from_redis. 2020-01-26 21:40:15 -08:00
Mateusz Mandera 859bde482d auth: Implement server side of desktop_flow_otp. 2020-01-26 21:40:15 -08:00
Mateusz Mandera 8d987ba5ae auth: Use tokens, with data stored in redis, for log_into_subdomain.
The desktop otp flow (to be added in next commits) will want to generate
one-time tokens for the app that will allow it to obtain an
authenticated session. log_into_subdomain will be the endpoint to pass
the one-time token to. Currently it uses signed data as its input
"tokens", which is not compatible with the otp flow, which requires
simpler (and fixed-length) token. Thus the correct scheme to use is to
store the authenticated data in redis and return a token tied to the
data, which should be passed to the log_into_subdomain endpoint.

In this commit, we replace the "pass signed data around" scheme with the
redis scheme, because there's no point having both.
2020-01-26 21:32:44 -08:00
Abhishek-Balaji 434e8d3104 home: Extract compute_show_invites_and_add_streams.
This extracts a function for computing show_invites and
show_add_streams, for better readability and testability.

This commit was substantially cleaned up by tabbott.
2020-01-25 23:41:08 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Mateusz Mandera 3f99985df3 auth: Require only subdomain and email in log_into_subdomain.
Tweaked by tabbott to add a comment explaining the reasoning.
2020-01-23 16:42:35 -08:00
Mateusz Mandera 90fe1bda02 auth: Extract create_response_for_otp_flow function. 2020-01-23 16:37:55 -08:00
Mateusz Mandera eafdefc5c9 auth: Extract finish_mobile_flow function. 2020-01-23 16:24:07 -08:00
Mateusz Mandera bce50ee652 auth: Use authenticate_remote_user in remote_user_jwt.
authenticate_remote_user already takes care of calling the authenticate
with the dummy backend. Also, return_data is not used and catching
DoesNotExist exception is not needed, as the dummy backend just returns
None if user isn't found.
2020-01-23 16:24:07 -08:00
Mateusz Mandera f00a852697 auth: Extract register_remote_user function. 2020-01-23 16:24:07 -08:00
Mateusz Mandera 2f38bc669f auth: Set request._email to user.delivery_email instead of user.email.
In other places where we set request._email, we set it to the
delivery_email, as that's more informative in orgs with hidden email
settings, where user.email will be useless.
2020-01-23 16:24:07 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Tlazypanda 30ee0c2a49 invitations: Improve experience around reactivating users.
Previously, if you tried to invite a user whose account had been
deactivated, we didn't provide a clear path forward for reactivating
the users, which was confusing.

We fix this by plumbing through to the frontend the information that
there is an existing user account with that email address in this
organization, but that it's deactivated.  For administrators, we
provide a link for how to reactivate the user.

Fixes #8144.
2020-01-13 18:30:51 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
Tim Abbott d8df1255d3 find_team: Send find team emails from the support address.
This is for consistency with the email's body, which claims replying
directly will work.
2020-01-08 21:55:34 -08:00
Mateusz Mandera 01288ede9e recipients: Remove bulk_get_recipients function and its uses.
With the recipient field being denormalized into the UserProfile and
Streams models, all current uses of bulk_get_recipients can be done more
efficient, by simply checking the .recipient_id attribute on the
appropriate objects.
2019-12-12 12:00:13 -08:00
Mateusz Mandera 4eb629e276 auth: Use config_error instead of JsonableError in remote_user_sso. 2019-12-11 16:40:20 -08:00
Mateusz Mandera e955bfde83 auth: Check that the backend is enabled at the start of remote_user_sso. 2019-12-11 16:35:18 -08:00
Mateusz Mandera 7ee54810a1 auth: Eliminate if/else block for PreregUser handling with/without SSO.
Both branches did very similar things, and the code is better having
common handling in all cases.
2019-12-10 20:16:21 +01:00
Nat1405 d5f005fd61 wildcard_mentions_notify: Add per-stream override of global setting.
Adds required API and front-end changes to modify and read the
wildcard_mentions_notify field in the Subscription model.

It includes front-end code to add the setting to the user's "manage
streams" page. This setting will be greyed out when a stream is muted.
The PR also includes back-end code to add the setting the initial state of
a subscription.

New automated tests were added for the API, events system and front-end.
In manual testing, we checked that modifying the setting in the front end
persisted the change in the Subscription model. We noticed the notifications
were not behaving exactly as expected in manual testing; see
https://github.com/zulip/zulip/issues/13073#issuecomment-560263081 .

Tweaked by tabbott to fix real-time synchronization issues.

Fixes: #13429.
2019-12-09 16:09:38 -08:00
Mateusz Mandera 1c5461663f users: Eliminate some unnecessary get_personal_recipient calls. 2019-12-09 15:24:35 -08:00
Mateusz Mandera 467833a974 streams: Eliminate some unnecessary get_stream_recipient calls. 2019-12-09 15:24:35 -08:00
Mateusz Mandera 74dd21c8fa register: Allow creating non-ldap users via social backends.
In configurations that use the ldap authentication backend and a social
backend, make it possible to create non-ldap users via the social backend.
2019-12-02 17:44:11 -08:00
Mateusz Mandera fcc91ae370 ldap: Disallow creating non-ldap accounts matching LDAP_APPEND_DOMAIN.
In configurations with LDAP_APPEND_DOMAIN, we don't want people creating
non-ldap accounts with emails matching the ldap domain.
So in the registration flow, if the email isn't found in LDAP, but
matches LDAP_APPEND_DOMAIN, we stop, rather than proceeding with account
creation. In case of emails not matching LDAP_APPEND_DOMAIN, we will
still continue to make a normal, non-ldap account.
2019-12-02 17:44:11 -08:00
Mateusz Mandera 82674b9b83 register: Improve handling of non-ldap users in LDAPPopulator configs.
The problem was that, for example, given a configuration of social
backend + LDAPPopulator, if a user that's not in ldap was being
registered, the Full Name field in the registration form would be
empty instead of getting prefilled with the name provided by the
social backend.

This fixes it - first we try to get the name from ldap. If that
succeeds, a form is created pre-filled with that name.  Otherwise, we
proceed to attempt to pre-fill with other means.

This also has a nice side effect of reorganizing most of the logic to
be more parallel between LDAP and other sources of name data.
2019-12-02 17:36:53 -08:00
Rohitt Vashishtha 68e93d2435 update-message: Use MentionData in the update_message_backend code.
This is a performance optimization, since we can avoid doing work
related to wildcard mentions in the common case that the message can't
have any.  We also add a unit test for adding wildcard mentions in a
message edit.
2019-12-02 12:12:35 -08:00
Mateusz Mandera 67b6179df2 ldap: Fix error while updating a user registered in multiple realms.
Previously, the LDAP code for syncing user data was not
multiple-realm-aware, resulting in errors trying to sync data for an
LDAP user present in multiple realms.

Tweaked by tabbott to add some extended comments.

Fixes #11520.
2019-11-21 11:13:31 -08:00
Mateusz Mandera 06c2161f7e auth: Use zxcvbn to ensure password strength on server side.
For a long time, we've been only doing the zxcvbn password strength
checks on the browser, which is helpful, but means users could through
hackery (or a bug in the frontend validation code) manage to set a
too-weak password.  We fix this by running our password strength
validation on the backend as well, using python-zxcvbn.

In theory, a bug in python-zxcvbn could result in it producing a
different opinion than the frontend version; if so, it'd be a pretty
bad bug in the library, and hopefully we'd hear about it from users,
report upstream, and get it fixed that way. Alternatively, we can
switch to shelling out to node like we do for KaTeX.

Fixes #6880.
2019-11-21 10:23:37 -08:00
Mateusz Mandera 0c2cc41d2e CVE-2019-18933: Fix insecure account creation via social authentication.
A bug in Zulip's new user signup process meant that users who
registered their account using social authentication (e.g. GitHub or
Google SSO) in an organization that also allows password
authentication could have their personal API key stolen by an
unprivileged attacker, allowing nearly full access to the user's
account.

Zulip versions between 1.7.0 and 2.0.6 were affected.

This commit fixes the original bug and also contains a database
migration to fix any users with corrupt `password` fields in the
database as a result of the bug.

Out of an abundance of caution (and to protect the users of any
installations that delay applying this commit), the migration also
resets the API keys of any users where Zulip's logs cannot prove the
user's API key was not previously stolen via this bug.  Resetting
those API keys will be inconvenient for users:

* Users of the Zulip mobile and terminal apps whose API keys are reset
  will be logged out and need to login again.
* Users using their personal API keys for any other reason will need
  to re-fetch their personal API key.

We discovered this bug internally and don't believe it was disclosed
prior to our publishing it through this commit.  Because the algorithm
for determining which users might have been affected is very
conservative, many users who were never at risk will have their API
keys reset by this migration.

To avoid this on self-hosted installations that have always used
e.g. LDAP authentication, we skip resetting API keys on installations
that don't have password authentication enabled.  System
administrators on installations that used to have email authentication
enabled, but no longer do, should temporarily enable EmailAuthBackend
before applying this migration.

The migration also records which users had their passwords or API keys
reset in the usual RealmAuditLog table.
2019-11-21 10:23:37 -08:00
Tim Abbott 1fe4f795af settings: Add notification settings checkboxes for wildcard mentions.
This change makes it possible for users to control the notification
settings for wildcard mentions as a separate control from PMs and
direct @-mentions.
2019-11-20 16:58:46 -08:00
Matheus Melo 39bd565941 test_realm: Automate tests for invalid integer values in Realm.
Fixes #13362.
2019-11-18 15:21:11 -08:00
Matheus Melo a890652bfd realm: Use check_int_in validator for enum format views.
This lets us remove some ugly semi-duplicated code.

Modified by tabbott to include some additional endpoints in the
changes.
2019-11-18 15:21:00 -08:00
Matheus Melo 31558cb8b9 decorator: Extract require_user_group_edit_permission.
We move the check that the user is a member or admin inot this
decorator.

This name better communicates that this may do other checks beyond
just verifying the policy.
2019-11-18 15:13:29 -08:00
Matheus Melo 21ed834101 decorator: Extract OrganizationAdministratorRequired common exception.
This eliminates significant code duplication of error messages for
situations where an organization administrator is required.
2019-11-18 15:10:56 -08:00
Tim Abbott 4e421ebe12 scripts: Move inline-email-css from tools to scripts.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
2019-11-15 17:39:42 -08:00
Tim Abbott 0e317a8773 registration: Use delivery_email in logging code path.
This is essentially an assertion failure code path, so it doesn't
really matter, but it seems best to use the value that's the cause of
the problem here.
2019-11-15 17:18:19 -08:00
Tim Abbott 7935b3cbd7 archive: Fix calculation of gravatar URL.
This is mostly dead code right now, but in case we revive it, I found
this error in an audit.
2019-11-15 17:14:44 -08:00
Tim Abbott df90533eef home: Switch invalid pointer logging to by user ID. 2019-11-15 17:07:52 -08:00
Tim Abbott 093e9394dd find_account: Use delivery_email when searching emails.
Previously, this code wouldn't work with
EMAIL_ADDRESS_VISIBILITY_ADMINS.
2019-11-15 17:07:52 -08:00
Anders Kaseorg cafac83676 request: Tighten type checking on REQ.
Then, find and fix a predictable number of previous misuses.

With a small change by tabbott to preserve backwards compatibility for
sending `yes` for the `forged` field.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:35:55 -08:00
Anders Kaseorg b0a7b33f9b push_notifications: Declare token of type str, not bytes.
Declaring a CharField of type bytes made no sense.

Signed-off-by: Anders Kaseorg <andersk@zulipchat.com>
2019-11-12 23:21:20 -08:00
Gloria Elston f8855ca179 api: Remove legacy emoji reactions endpoint.
The original/legacy emoji reactions endpoints made use of HTTP PUT and
didn't have an API that could correctly handle situations where the
emoji names change over time.  We stopped using the legacy endpoints
some time ago, so we can remove them now.

This requires straightforward updates to older tests that were still
written against the legacy API.

Fixes #12940.
2019-11-12 13:07:06 -08:00
Tim Abbott ddd1a0eb00 actions: Convert do_delete_messages to take a Realm.
The function only used the user's realm anyway, so this is a cleaner
API.

This should also make it more convenient to permanently delete
messages manually, since one doesn't have to fetch a random user in
the realm in order to delete a message using the management shell.

No functional change.
2019-11-12 12:20:31 -08:00
Mateusz Mandera ed40d37e44 ldap: Fix realm_creation=True registration flow.
When creating realm with the ldap backend, the registration flow didn't
properly handle some things - the user wouldn't be set as realm admin,
initial subscriptions and messages weren't created, and the redirect
wasn't happening properly in the case of subdomains.
2019-11-08 14:01:45 -08:00
Mateusz Mandera 9ca26e91e0 ldap registration: Refactor, renaming auth_result to user_profile. 2019-11-08 14:01:45 -08:00
Mateusz Mandera fc4ad44dae registration: Refactor, reordering some conditional blocks. 2019-11-08 14:01:45 -08:00
Tim Abbott 47cf99a970 realm: Prevent setting invalid enum policy values.
As discussed in the comment, ideally these checks should be added
completely automatically, rather than needing to be manually added
every time we add a new setting.  But hopefully the example code for
all of the similar enums that this provides will at least provide some
help.
2019-11-03 17:11:56 -08:00
Tim Abbott 4066b58faa send_message: Remove long-obsolete comment about @require_login.
All of our view functions work this way, so it doesn't need a special
comment.
2019-11-03 16:52:28 -08:00
Matheus Melo c96762b7a9 settings: Add setting for who can edit user groups.
Fixes #12380.
2019-11-03 16:45:13 -08:00
Mateusz Mandera 5aded51b73 register: Pre-populate Name in social backend flow.
By adding some additional plumbing (through PreregistrationUser) of the
full_name and an additional full_name_validated option, we
pre-populate the Full Name field in the registration form when coming
through a social backend (google/github/saml/etc.) and potentially skip
the registration form (if the user would have nothing to do there other
than clicking the Confirm button) and just create the account and log
the user in.
2019-11-03 16:15:48 -08:00
Mateusz Mandera 34a540bacb context: Rename social_backends to external_authentication_methods.
The main purpose of this is to make that name change happen in
/server_settings. external_authentication_methods is a much better, more
descriptive name than social_backends from API perspective.
2019-11-03 15:55:44 -08:00
Mateusz Mandera 339f742578 test_signup: Add test for a non-covered case in accounts_register. 2019-11-03 15:51:19 -08:00
Mateusz Mandera 2e1529e879 registration: Fix logic disabling password requirement in the form.
This small block of code was over-indented. It should be run in this
part of the function unconditionally, not inside an "else" block.
We obviously want it to run regardless of whether
request.POST.get('from_confirmation')
is True or not.
2019-11-01 16:59:06 -07:00
Mateusz Mandera 29314f3195 api: Remove unused /get_auth_backends endpoint.
This legacy endpoint was designed for the original native Zulip mobile
apps, which were deprecated years ago in favor of the React Native
app.

It was replaced by /server_settings for active use years ago, so it's
safe to remove it now.
2019-11-01 12:30:59 -07:00
Hemanth V. Alluri c1370547d5 events: Only send bot_type for bots and thus remove the for_api param. 2019-10-29 15:41:35 -07:00
Hemanth V. Alluri 5b3e346369 users: Send custom profile fields with the /profile endpoint. 2019-10-29 15:41:35 -07:00
Mateusz Mandera b870816a75 saml: Sanity-check configuration in both login and signup codepaths. 2019-10-28 15:11:19 -07:00
Mateusz Mandera db29fcbbc4 auth: Add social_backends to /server_settings. 2019-10-28 15:11:19 -07:00
Mateusz Mandera 892d25faa1 auth: Change SAML login url scheme, enabling multiple IdP support.
The url scheme is now /accounts/login/social/saml/{idp_name} to initiate
login using the IdP configured under "idp_name" name.

display_name and display_logo (the name and icon to show on the "Log in
with" button) can be customized by adding the apprioprate settings in
the configured IdP dictionaries.
2019-10-28 15:09:42 -07:00
Mateusz Mandera 8c065d1fcd ldap: Ensure django_to_ldap_username returns username that is in ldap.
This changes the way django_to_ldap_username works to make sure the ldap
username it returns actually has a corresponding ldap entry and raise an
exception if that's not possible. It seems to be a more sound approach
than just having it return its best guess - which was the case so far.
Now there is a guarantee that what it returns is the username of an
actual ldap user.

This allows communicating to the registration flow when the email being
registered doesn't belong to ldap, which then will proceed to register
it via the normal email backend flow - finally fixing the bug where you
couldn't register a non-ldap email even with the email backend enabled.

These changes to the behavior of django_to_ldap_username require small
refactorings in a couple of other functions that call it, as well as
adapting some tests to these changes. Finally, additional tests are
added for the above-mentioned registration flow behavior and some
related corner-cases.
2019-10-25 12:14:51 -07:00
Hemanth V. Alluri 1946692f9a users: Refactor get_members_backend endpoint to use get_raw_user_data.
Modify the get_raw_user_data method for use by the /users API endpoint
and then modify the /users endpoint to use it.
2019-10-23 14:50:26 -07:00
Tim Abbott d364891894 ldap: Fix password prompt when configured only to populate data.
Previously, the logic for determining whether to provide an LDAP
password prompt on the registration page was incorrectly including it
if any LDAP authentication was backend enabled, even if LDAP was
configured with the populate-only backend that is not responsible for
authentication (just for filling in name and custom profile fields).

We fix this by correcting the conditional, and add a test.

There's still follow-up work to do here: We may still end up
presenting a registration form in situations where it's useless
because we got all the data from SAML + LDAP.  But that's for a future
issue.

This fixes a bug reported in #13275.
2019-10-17 14:46:46 -07:00
Rafid Aslam 718b70ec8b Rename `not_subscribed key` to `not_removed` in users/me/subscriptions.
Rename `not_subscibed_key` to `not_removed` in
`users/me/subscriptions` DELETE response.

Fixes #13277.
2019-10-13 10:30:34 +07:00
Mateusz Mandera 4dc3ed36c3 auth: Add initial SAML authentication support.
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.

This PR is a collaboration with Tim Abbott.

Fixes #716.
2019-10-10 15:44:34 -07:00
Vishnu KS cd06e0ab79 tests: Remove /messages/{message_id} from buggy endpoints. 2019-10-07 11:45:11 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Tim Abbott c869a3bf82 upload: Fix browser caching of uploads with local uploads backend.
Apparently, our change in b8a1050fc4 to
stop caching responses on API endpoints accidentally ended up
affecting uploaded files as well.

Fix this by explicitly setting a Cache-Control header in our Sendfile
responses, as well as changing our outer API caching code to only set
the never cache headers if the view function didn't explicitly specify
them itself.

This is not directly related to #13088, as that is a similar issue
with the S3 backend.

Thanks to Gert Burger for the report.
2019-10-01 15:15:17 -07:00
Ivan Mitev 16c9d63056 emoji: Fix sort order of emoji choices.
The previous code for ensuring the sort order of emoji choices was
correct relied on an OrderedDict structure, which isn't guaranteed to
be preserved when passed to the frontend via JSON (in fact, it isn't,
since we converted the way page_params is passed to use
sort_keys=True).  Switch it to a list of dictionaries to correct this.

Fixes #13220.
2019-10-01 13:54:55 -07:00
Mateusz Mandera 4166c901ef do_update_user_custom_profile_data: Rename to ..._if_changed.
This adds clarity to the fact that the function no longer does
anything if the field values haven't changed.
2019-10-01 13:52:43 -07:00
Tim Abbott bf088519a7 api: Add avatar URL to the GET /profile endpoint.
This endpoint is legacy in a lot of ways, but it seems reasonable that
it should have these data.
2019-09-27 12:07:03 -07:00
Anders Kaseorg 9339402541 messages: Sort IDs in the by_streams query deterministically.
This fixes a test flake after upgrading SQLAlchemy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Yago González e1fbf6dddb actions: Make do_regenerate_api_key return the new key.
This way, the new API key can be fetched without needing to read it from
the UserProfile object.
2019-09-19 13:03:56 -07:00
Anders Kaseorg fe7d814e8d team: Move contributors_list into page_params.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 7494f1600c templates: Move page_params from an inline script to the <body> dataset.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 780ecb672b CVE-2019-16216: Fix MIME type validation.
* Whitelist a small number of image/ types to be served as
  non-attachments.
* Serve the file using the type that we validated rather than relying
  on an independent guess to match.

This issue can lead to a stored XSS security vulnerability for older
browsers that don't support Content-Security-Policy.

It primarily affects servers using Zulip's local file uploads backend
for servers running Ubuntu 16.04 Xenial or newer; the legacy local
file upload backend for (now EOL) Ubuntu 14.04 Trusty was not affected
and it has limited impact for the S3 upload backend (which uses an
unprivileged S3 bucket domain to serve files).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-11 15:46:36 -07:00
Mateusz Mandera d70e1bcdb7 settings: Add FAKE_EMAIL_DOMAIN setting.
Fixes #9401.

This adds a FAKE_EMAIL_DOMAIN setting, which should be used if
EXTERNAL_HOST is not a valid domain, and something else is needed to
form bot and dummy user emails (if email visibility is turned off).
It defaults to EXTERNAL_HOST.

get_fake_email_domain() should be used to get this value. It validates
that it's correctly set - that it can be used to form valid emails.

If it's not set correctly, an exception is raised. This is the right
approach, because it's undesirable to have the server seemingly
peacefully operating with that setting misconfigured, as that could
mask some hidden sneaky bugs due to UserProfiles with invalid emails,
which would blow up the moment some code that does validate the emails
is called.
2019-08-30 14:59:00 -07:00
Anders Kaseorg c3e395b7d8 search: Reimplement ts_locs_array in pure PostgreSQL.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:59:12 -07:00
Yashashvi Dave e05429b6a9 custom fields: Add default external account custom fields. 2019-08-28 15:35:53 -07:00
Tim Abbott 73258efb16 development: Accept ToS for create user buttons.
We added default ToS for the development environment a few months
back; as a side effect, we now need to accept ToS when going through
the development environment registration flow, including for our
one-click account creation buttons.
2019-08-26 13:55:55 -07:00
Yashashvi Dave ed5cb6c28b custom profile fields: Extract field validation function. 2019-08-25 20:59:08 -07:00
Mohit Gupta e5482adec0 search: Add streams:public to search entire history of public streams.
Add ability to search entire message history of all public streams at
once. It includes all subscibed, non subscribed public streams messages
and even historical public stream messages sent before user had joined
an organization or stream.

Fixes #8859.
2019-08-22 13:40:49 -07:00
Hemanth V. Alluri d73a37726d bots: Allow incoming webhook bots to be configured via /bots.
Without disturbing the flow of the existing code for configuring
embedded bots too much, we now use the config_options feature to
allow incoming webhook type bot to be configured via. the "/bots"
endpoint of the API.
2019-08-20 17:00:48 -07:00
Hemanth V. Alluri 94c351ead4 bot_config: Have check_valid_bot_config also take the bot_type.
This is a prep commit to allow us to validate user provided bot
config data using the same function for incoming webhook type
bots alongside embedded bots (as opposed to creating a new
function just for incoming webhook bots).
2019-08-20 16:44:56 -07:00
Pragati Agrawal 48efd46bc6 realm_logo: Remove redundant `realm_logo_url` function.
This function is nothing but a redundant layer of call to
`get_realm_logo_url`.
2019-08-20 12:07:20 -07:00
Tim Abbott 44db59bbe0 openapi: Fix handling of parameters passed via the URL/path.
Previously, our OpenAPI documentation validation was failing for some
endpoints because it didn't account for the `in: path` type of
parameter, resulting in a mismatch between what was declared via REQ
and what was declared in the OpenAPI docs.

We fix this by excluding the path type parameters in both places from
what's considered by documentation using the `path_only` flag.

I doubt this is the correct long-term fix; in particular, I don't
think we're actually running the validators for these path-only
parameters.  The examples that exist today are all IDs with validators
for being non-negative numbers, but longer-term I think we'll want to
do something different (possibly at the REQ layer, see the TODO).
2019-08-19 15:06:08 -07:00
Priyank Patel 3680393b47 messages: Support passing user ID for stream operator.
ok_to_include_history fuction was updated to expect stream ID.

Fixes part of #9474.
2019-08-17 11:19:12 -07:00
Priyank Patel 0307b6d684 refactor: Make get_stream_name_from_narrow return stream.
This function was used only once in exclude_muting_conditions where
it returned stream name so the function to fetch stream id.
Since we exepect the narrow to also include stream id we refactor it to
return a stream object and since we use get_stream_by_narrow_operand_access_unchecked
we don't need to worry about handling cases where stream id is passed since
the function handles it.
2019-08-17 11:10:00 -07:00
Priyank Patel 463ecb375f refactor: Add get_stream_by_narrow_operand_access_unchecked.
This let's us clean up the linter that excludes the use of get_stream
and by adding the access_unchecked in the name we make it clear that
it should be used with caution.

Refactoring idea by Tim Abbott.
2019-08-17 11:10:00 -07:00
Priyank Patel 1f8f8867cd message_fetch: Rename handle_user_ids_supported_operators.
This renames handle_user_ids_supported_operators to
handle_operators_supporting_id_based_api.
2019-08-17 11:10:00 -07:00
Priyank Patel 1edde4a989 Rename user_id(s)_supported_operators -> operators_supporting_id(s). 2019-08-17 11:10:00 -07:00
Priyank Patel 99450f336c messages: Extract mypy typing for narrow list to variable.
Makes it obvious and readable and as an added bonus take up less space.

Note, some types used Iterable instead of List that were change
to used List since the narrow_paramter converter return a List.
2019-08-17 11:10:00 -07:00
Tim Abbott 8ac8247dbe data export: Add limits on exporting large organizations.
It seems possible that attempting to export large organizations could
result in high resource consumption that justifies having a technician
manage the exports manually.
2019-08-12 18:21:09 -07:00
Wyatt Hoodes 896886b412 realm_export: Add endpoint for export deletion. 2019-08-12 17:51:46 -07:00
Anders Kaseorg e0cf3d0e76 zerver: Accept HEAD requests wherever GET requests are accepted.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Anders Kaseorg 72655611ce requirements: Use maintained fork django-sendfile2 of django-sendfile
The original seems to be unmaintained
(johnsensible/django-sendfile#65).  Notably, this fixes a bug in the
filename parameter, which perviously showed the Python 3 repr of a
byte string (johnsensible/django-sendfile#49).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:40:08 -07:00
Anders Kaseorg bb5507009f camo: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
meiomorphism 1af72a2745 subscriptions: Avoid sending unneeded subscriber information.
The `users/me/subscriptions` endpoint accidentally started returning
subscriber information for each stream.  This is convenient, but
unnecessarily costly for those clients which either don't need it
(most API apps) or already acquire this information via /register
(including Zulip's apps).

This change removes that data set from the default response.  Clients
which had come to rely on it, or would like to rely on it in future,
may still access it via an additional documented API parameter.

Fixes #12917.
2019-08-09 16:03:21 -07:00
Anders Kaseorg 13a2ac7b8e request: Remove ExtractRecipients type safety hole on REQ.
It was allowing us to get away with wrong types on a few functions:
`check_send_typing_notification` and `send_notification_backend` can be
(and are) called with a list of `int` as `notification_to`, not just a
list of `str`.

The problem it was working around already had a better solution using
the dummy `type` argument.  Use that.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:26:59 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Rishi Gupta b37c6a8644 custom profile: Update a few error messages. 2019-08-05 20:59:47 -07:00
Rishi Gupta 8167b5b645 email: Remove special integration doc behavior when gateway not set.
The current code looks like it's trying to redirect /integrations/doc/email
to /integrations when EMAIL_GATEWAY_PATTERN is not set.

I think it doesn't currently do this. The test for that pathway has a bug:
self.get_doc('integrations/doc-html/email', subdomain='zulip') needs a
leading slash, and putting the slash back in results in the test failing.

This redirection is not really desired behavior -- better is to
unconditionally show that the email integration exists, and just point the
user to https://zulip.readthedocs.io/en/latest/production/email-gateway.html
(this is done in a child commit).
2019-08-05 17:33:05 -07:00
Wyatt Hoodes b005c7567a messages: Fix typing errors.
We remove a redundant term conversion test, as well as
add appropriate typing for the `anchor` variable in
`get_messages_backend`.
2019-07-31 12:42:29 -07:00
Wyatt Hoodes f22b2c7e9d archive: Fix typing for prev_sender variable.
We know that via the `AbstractMessage` class that `sender`
is of the type `UserProfile`.  We type this as `Optional`
to tell mypy that the operands to the right of the first
`or` can indeed be evaluated within the following `for` loop.
2019-07-29 15:23:10 -07:00
Wyatt Hoodes 99bd55917e zephyr: Fix typing for cred parameter. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 0834514668 auth: Fix typing for authenticate_remote_user. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes bbbea9ec87 events: Rewrite system for managing realm exports.
This feature is intended to cover all of our ways of exporting a
realm, not just the initial "public export" feature, so we should name
things appropriately for that goal.

Additionally, we don't want to include data exports in page_params;
the original implementation was actually buggy and would have.
2019-07-26 16:38:52 -07:00
Wyatt Hoodes ef02de4834 public_export: Add endpoint for returning all REALM_EXPORTED objects. 2019-07-26 15:52:02 -07:00
Wyatt Hoodes 266ed9ff75 public_export: Add acting_user to RealmAuditLog object.
An acting_user attribute is required to correctly reference who
initiated a realm_exported event.
2019-07-26 15:49:47 -07:00
Hemanth V. Alluri ede73ee4cd streams: New stream messages should be auto marked as read by creator.
When a person creates a new realm, they'll likely want to create a
bunch of initial streams at once. When doing so, it could be annoying
to have to mark all of the new stream notification messages as read.

Thus to make this process smoother, we should automatically mark
the messages generated by the Notification Bot in the notifications
(announcements) stream, as well as in the newly created stream itself
as read by the stream creator.

Fixes #12765.
2019-07-26 15:10:09 -07:00
Anders Kaseorg 2b33822de9 storage: Stop using django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg fd7803e7f4 settings: Unset STATIC_ROOT in development.
Django’s default FileSystemFinder disallows STATICFILES_DIRS from
containing STATIC_ROOT (by raising an ImproperlyConfigured exception),
because STATIC_ROOT is supposed to be the result of collecting all the
static files in the project, not one of the potentially many sources
of static files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Tim Abbott 47d80977ac plans: Make usable in the development environment for testing. 2019-07-22 17:43:43 -07:00
Mateusz Mandera be0a024355 integration docs: Document new email integration behaviors.
Tweaked by tabbott for cleaner, more precise English.
2019-07-22 15:52:08 -07:00
Harshit Bansal bf14a0af4d auth: Migrate google auth to python-social-auth.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend.  These are:

* GoogleMobileOauth2Backend, the ancient code path for mobile
  authentication last used by the EOL original Zulip Android app.

* The `finish_google_oauth2` code path in zerver/views/auth.py, which
  was the webapp (and modern mobile app) Google authentication code
  path.

This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
2019-07-21 20:51:34 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Rishi Gupta 38eeb3e183 notification bot: Add constant for STREAM_EVENTS_NOTIFICATION_TOPIC. 2019-07-20 14:39:15 -07:00
Rishi Gupta c9f194be99 notification bot: Update message for stream name changes.
Also properly internationalizes string formatting arguments for the stream
created message.
2019-07-20 14:39:15 -07:00
Priyank Patel b338fd130e messages: Support passing user IDs for group-pm-with operator. 2019-07-17 16:09:12 -07:00
Priyank Patel c067c155aa messages: Support passing user ID to sender operator. 2019-07-17 16:09:12 -07:00
Anders Kaseorg 07f93d58a3 views: Remove unused APIView class.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-17 16:03:35 -07:00
Anders Kaseorg 5ec2e4add4 home: Remove handler for old compile-handlebars-templates error file.
As of commit 8c199fd44c (#12667) this
file is no longer generated.  Handlebars compile errors are raised as
webpack errors.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-17 16:03:35 -07:00
Rishi Gupta 5cf304f8f9 notification bot: Tighten message for being added to streams. 2019-07-13 15:53:36 -07:00
David Wood 9bace3f2cd notifications: Allow only notifiable in unread count.
This commit adds a new setting to the user's notification settings that
will change the behaviour of the unread count in the title bar and
desktop application.

When enabled, the title bar will show the count of unread private messages
and mentions. When disabled, the title bar will act as before, showing
the total number of unread messages.

Fixes #1736.
2019-07-13 15:49:04 -07:00
Priyank Patel 73b19672c3 message_fetch: Use user IDs for supported operators.
The approach taken here is basically use user IDs in operator that
support it when sending the request for fetching the messages
(see comments in code for more details).
2019-07-13 11:35:37 -07:00
Rishi Gupta 0f3c2748dd notification bot: Update stream announcement message. 2019-07-11 15:22:36 -07:00
Puneeth Chaganti 6fdd8fb4a1 documentation: Change open graph metadata for /integrations. 2019-07-11 14:23:25 -07:00
Rishi Gupta c3adafbee9 notification bot: Remove check for existence in add_subscriptions_backend.
We implicitly assume settings.NOTIFICATION_BOT is not None just a few lines
above, in
sender = get_system_bot(settings.NOTIFICATION_BOT)
notifications.append(
    internal_prep_private_message(
        realm=user_profile.realm,
        sender=sender,
        ...
2019-07-11 14:18:21 -07:00
Rishi Gupta 6c8569d99f notification bot: Add stream creator to initial stream message.
Also removes the "Welcome", since it's cleaner to have this be strictly a
notification.
2019-07-11 14:16:56 -07:00
Rishi Gupta 5400eb62be notification bot: Inline prep_stream_welcome_message.
This will make it easier to have access to the stream creator.

The indirection also isn't really adding anything, especially given that the
announce message is inlined just above.
2019-07-11 14:16:56 -07:00
Yashashvi Dave d7ee2aced1 models: Add `external_account` in custom profile field types.
Add new custom profile field type, External account.
External account field links user's social media
profile with account. e.g. GitHub, Twitter, etc.

Fixes part of #12302
2019-07-09 17:21:54 -07:00
sameerchoubey 205657b3cf api: Remove spammy json_success content for edit_user_group.
These values were unnecessary translated strings not displayed
anywhere, and were not consistent with Zulip's API standards.
2019-07-09 13:04:47 -07:00
Hemanth V. Alluri 2738b09044 openapi: Remove a few "buggy" endpoints in the validation test.
By importing a few view modules in the validation test itself we
can remove a few endpoints which were marked as buggy. What was
happening was that the view functions weren't imported and hence
the arguments map was not filled. Thus the test complained that
there was documentation for request parameters that seemed to be
missing in the code. Also, for the events register endpoint, we
have renamed one of the documented request parameters from
"stream" to "topic" (the API itself was not modified though).

We add a new "documentation_pending" attribute to req variables
so that any arguments not currently documented but should be
documented can be properly accounted for.
2019-07-08 12:34:31 -07:00
Mohit Gupta 45f87ff44b search: Fix to allow access to group PM containing deactivated users.
Fixes: #12593.
2019-07-03 15:30:19 -07:00
Mohit Gupta c971576b00 search: Refactor by_pm_with to use user profiles as common path.
Use UserProfiles instead of emails to fetch recipient objects for
narrowing; this is cleaner as it avoids unnecessary parsing and
unparsing.  We just map ids/emails operand to user profiles and then
use common code from there.

Fixes #12601.
2019-07-03 15:29:08 -07:00
Anders Kaseorg 5e97e58df3 webhooks/freshdesk/doc.md: Remove unescape_rendered_html kludge.
This reverts commit f476ec7fac (#10312)
and replaces it with a proper fix using Jinja2 raw blocks.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 15:05:45 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
Puneeth Chaganti c6fcf36c8d documentation: Add open graph metadata for integrations pages. 2019-06-26 16:32:13 -07:00
Hemanth V. Alluri cd9c1bfb9d devtools: Update the send_all feature of the integrations tool for headers.
Now that we have a system for storing HTTP headers for each integration, we
should fix the send_all button. Previously, it used the same user entered
custom HTTP header (from the GUI) for all of the fixtures, but now we
automatically determine the header with the new system instead.
2019-06-26 10:35:16 -07:00
Hemanth V. Alluri e2549b3b84 webhooks: Replace headers parsing logic with a common source.
When parsing custom HTTP headers in the integrations dev panel, http
headers from fixtures system and the send_webhook_fixture_message
we now use a singular source of logic: standardize_headers which
will take care of converting a dictionary of input headers into a
standard form that Django expects.
2019-06-24 13:23:21 -07:00
Hemanth V. Alluri 720c866e06 devtools: Tweak the integrations tool to use http headers fixtures.
Now that we store HTTP headers in a way that is easy to retreive
by specifying the integration name and fixture name, we should
use it to pre-load the "Custom HTTP Headers" field in the
integrations dev panel.
2019-06-20 16:40:42 -07:00
Tim Abbott 4eb9d67b70 models: Extract get_human_admin_users function.
This function is an alternative to get_admin_users that we use in all
places where we explicitly want only human administrative users (not
administrative bots).  The following commits will rename
get_admin_users for better clarity.
2019-06-20 14:32:30 -07:00
Mohit Gupta db3d81613b decorator: Refactor @require_non_guest_human_user decorator.
Rename @require_non_guest_human_user to @require_member_or_admin.
This is a refactor commit prior to introduction of Administrator
Bots.
2019-06-18 17:11:58 -07:00
Priyank Patel d7b4de2348 messages: Add support for passing user IDs for pm-with clause.
We also document support for user IDs in the pm-with narrow operator.

Edited by tabbott to document on /api rather than in the /help page.

Fixes part of #9474.
2019-06-17 17:38:51 -07:00
Yashashvi Dave 0b4384f4cd custom profile fields: Extract function `validate_custom_field_data`. 2019-06-17 14:56:02 -07:00
Yashashvi Dave 8e269b4651 models: Rename notification to `enable_stream_audible_notifications`.
Rename notification property `enable_stream_sounds` to
`enable_stream_audible_notifications` to match with other
notification property patterns.

Fixes part of #12304
2019-06-12 16:24:51 -07:00
Yashashvi Dave f25e00bcaa zever/lib/validator: Rename function to `validate_choice_field_data`. 2019-06-06 22:36:16 -07:00
sahil839 5a130097bf settings: Add display setting for demoting inactive streams.
This adds a setting to control Zulip's default behavior of sorting to
bottom and graying out inactive streams.  The previous logic is still
the default "automatic", but this gives users more control.  See the
models.py comment for details.

Fixes #11524.
2019-06-03 23:07:56 -07:00
Tim Abbott 807e5c7a1a auth: Fix fetching personal API key with email addresses hidden.
This was a corner case missed in the main migration.
2019-06-03 15:55:07 -07:00
Wyatt Hoodes db69cdbcde public_export: Add support for deleting export after access.
The RealmAuditLog object ID was stored in the event sent to the
deferred_work queue as a means to update the row's extra_data field.
The extra_data field then stores the location of the export.
2019-05-31 22:54:27 -07:00
Eeshan Garg cecea75457 api_docs: Detect missing arguments in curl examples.
This commit adds automated tests that make sure that every curl
example command in our API docs has the '-X (POST|GET)' argument.

Fixes: #11927
2019-05-28 16:53:48 -07:00
Anders Kaseorg 082f23a659 authenticate: Remove default values for required parameters.
It is now the caller’s responsibility to check that realm is not None.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:47:22 -07:00
Anders Kaseorg 725582850f login_or_register_remote_user: Remove unused invalid_subdomain parameter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:47:22 -07:00
Wyatt Hoodes 46790b3280 public_export.py: Reorder the creation of the RealmAuditLog object.
This reordering was originally made with regard to the delete after
access feature for the public export.  However, this reordering is
more correct overall, i.e., the object should be created before the
event pertaining to the object is sent.
2019-05-27 20:07:28 -07:00
Yashashvi Dave 3e50ed2075 org settings: Add organization profile preview option.
This should make it convenient and obvious how verify that their
organization profile looks nice after being markdown-rendered.

Fixes #12105.
2019-05-21 17:53:34 -07:00
Mayank Madan d5e61e20e5 dev tooling: Add "create new user" and "create new realm" buttons.
Significantly tweaked by tabbott to clean up and expand the tests.

Fixes: #6018.
2019-05-21 15:03:37 -07:00
Tim Abbott 6a42280e31 auth: Fix devlogin "All realms" view.
This was apparently accidentally broken (making it 500) by the
refactoring in 9efda71a4b.
2019-05-21 14:46:15 -07:00
Joshua Pan c28c301506 org_settings: Display delivery_email to admins always is admin UI.
Mostly rewritten by Tim Abbott to ensure it correctly implements the
desired security model.

Administrators should have access to users' real email address so that
they can contact users out-of-band.
2019-05-20 18:13:31 -07:00
Tim Abbott 07856ad648 api: Override client_gravatar for EMAIL_ADDRESS_VISIBILITY_ADMINS.
Clients won't have access to user email addresses, and thus won't be
able to compute gravatars.

The tests for this are a bit messy, in large part because our tests
for get_events call subsections of it, rather than the main function.
2019-05-20 18:13:31 -07:00
Hemanth V. Alluri 1a12e112d9 devtools: Completely overhaul the frontend for the integrations devtool.
This commit also adds a small functionality change where the results of
each webhook fixture message sent is now displayed to the user.

With a small tweak by tabbott to fix a styling bug.

Fixes #12122.
2019-05-20 12:36:15 -07:00
Hemanth V. Alluri ef98211f68 devtools: Add support for non-json fixtures for the integrations tool.
Note: If you're going to send fixtures which are not JSON or of the
text/plain content type, make sure you set the correct content type
in the custom headers.

E.g. For the wordpress fixtures the "Content-Type" should be set to
"application/x-www-form-urlencoded".
2019-05-20 12:31:52 -07:00
Hemanth V. Alluri 8214d65336 devtools: Add support for send_all in the integrations dev panel.
Using this feature a reviewer can easily send and view all fixture
messages for any given integration - with only JSON fixtures that is.
2019-05-20 12:29:10 -07:00
Hemanth V. Alluri 2bd9c8cb42 devtools: Add custom HTTP headers support to the integrations dev panel.
This commit introduces a simple field where the user can now specify custom
HTTP headers. This commit does not introduce an improved system for storing
HTTP headers as fixtures - such a change would modify both the existing unit
tests as well as this devtool.
2019-05-17 11:50:53 -07:00
Hemanth V. Alluri bae8295c52 devtools: Add integrations dev panel.
This commit adds a new developer tool: The "integrations dev panel"
which will serve as a replacement for the send_webhook_fixture_message
management command as a way to test integrations with much greater ease.
2019-05-15 13:07:44 -07:00
vinitS101 611f1f8fd2 left_sidebar: Add "+Add streams" to bottom of streamlist.
Added a new button at the bottom of the stream list which redirects
users to '/#streams/all' where they can create new streams or subscribe
to new streams.
The button is not visible to guests.

Fixes #11642.
2019-05-14 16:12:51 -07:00
Yashashvi Dave 3efead6f5f /json/realm: Return error on not-avaliable video chat provider value.
This adds some missing error handling from the video chat provider
code path.
2019-05-13 12:07:44 -07:00
Yashashvi Dave cb85ca8601 models: Alter video_chat_provider field type to integer.
Migration rewritten by tabbott because it did not work.
2019-05-13 12:02:28 -07:00
Yashashvi Dave d7f9b7eea3 models: Add dict for video chat providers in Realm model. 2019-05-13 11:34:34 -07:00
Tim Abbott 45305c93ae auth: Fix Apache SSO port number in confirmation links.
We had a report in the thread around
https://chat.zulip.org/#narrow/stream/31-production-help/topic/Apache-based.20SSO/near/741013
that confirmation links were taking the user to the /register form on
the Apache server, which of course doesn't work because the Apache
server architecture we have is intended to only serve a single
endpoint, /accounts/login/sso, and not any static assets (etc.).

This manifested as users getting a broke page with a bunch of JS
errors about missing static assets when trying to sign up for an
account.  The right fix is to ensure that we serve these confirmation
links (and maybe in the future, redirects) to the nginx server.
2019-05-13 11:14:41 -07:00
Roman Godov a50824e031 models: Rename Subscription.in_home_view field to is_muted.
This renames Subscription.in_home_view field to is_muted, for greater
clarity as to what it does just from seeing the setting name, without
having to look it up.

Also disabled an obsolete test_migrations test.

Fixes #10042.
2019-05-12 22:08:10 -07:00
Pragati Agrawal a838de63d8 settings: Add option for fixed/fluid width.
This adds a new option for "fluid width" under `Display settings` section
of SETTINGS/DISPLAY SETTINGS tab.

Fixes: #11844.
2019-05-09 13:57:57 -07:00
Harshit Bansal b553507412 subscriptions: Migrate notification setting defaults model.
This commit migrates the Subscription's notification fields from a
BooleanField to a NullBooleanField where a value of None means to
inherit the value from user's profile.

Also includes a migrations to set the corresponding settings to None
if they match the user profile's values. This migration helps us in
getting rid of the weird "Apply to all" widget that we offered on
subscription settings page.

The mobile apps can't handle None appearing as the stream-level
notification settings, so for backwards-compatibility we arrange to
only send True/False to the mobile apps by applying those defaults
server-side.  We introduce a notification_settings_null value within a
client_capabilities structure that newer versions of the mobile apps
can use to request the new model.

This mobile compatibility code is pretty effectively tested by the
existing test_events tests for the subscriptions subsystem.
2019-05-08 17:45:10 -07:00
vinitS101 18a424be79 uploads: Remove unusable UI elements if file uploading is disabled.
If MAX_FILE_UPLOAD_SIZE is set to 0, then UI elements like the upload
icon in the compose and message edit UI and "Attachments" menu in
"/#settings" are not displayed.
A different error message is also displayed if a user tries to drag and
drop or paste a file into the compose message box.

Fixes #12152.
2019-05-08 17:10:07 -07:00
Rishi Gupta d2b32749cc audit log: Log acting_user for do_deactivate_realm.
Currently there's no way to tell the difference between "a server admin
deactivated a realm due to it being spammy" vs "a realm admin deactivated
the realm".
2019-05-08 15:09:48 -07:00
Anders Kaseorg 9efda71a4b get_realm: raise DoesNotExist instead of returning None.
This makes the implementation of `get_realm` consistent with its
declared return type of `Realm` rather than `Optional[Realm]`.

Fixes #12263.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-06 21:58:16 -07:00
David Wood 34d810aac3 settings: Migrate to create_stream_policy structure.
This commit replaces the `create_stream_by_admins_only` setting with a
new `create_stream_policy` setting, which mirroring the structure of
the existing `invite_to_stream_policy`.

This is important preparation for migrating the waiting period feature
to be its own independent setting.

Fixes #12236.
2019-05-06 16:27:55 -07:00