Commit Graph

558 Commits

Author SHA1 Message Date
Anders Kaseorg 1f565a9f41 timezone: Use standard library datetime.timezone.utc consistently.
datetime.timezone is available in Python ≥ 3.2.  This also lets us
remove a pytz dependency from the PostgreSQL scripts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
sahil839 2fe8971204 populate_db: Change role of 'desdemona' to realm owner.
Modified by tabbott to update PROVISION_VERSION.
2020-06-01 15:22:27 -07:00
clarammdantas f0c85c5796 registration: Show error if invalid key is posted on /accounts/register.
If the key paramenter on POST isn't correct we won't be
able to find the confirmation object, which will lead to
an exception. To deal with it more gracefully, we are
catching the exception and redirecting to the
confirmation_link_expired_error page.
2020-05-27 15:37:16 -07:00
clarammdantas edcf4f0ea2 invitations: Revoke remaining invitations after user registers.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.

We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.

Fixes: #12629

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
clarammdantas 923c5e3063 test_signup: Check if user is already registered.
This tests if a user, that is already registered, is
redirected to the login page when they click on an
invitation.

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
clarammdantas 8869f05c20 test_signup: Attach prereg_user to confirmation obj.
Tests attached a UserProfile to confirmation objects,
which is not very valid as this is the only place
where this is done. Now we attach PreregUser to
the confirmation object, making the tests correct.

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
arunikaydav42 4680d504de invite: Fix invitations removed from list too soon.
On invitations panel, invites were being removed when
the user clicked on invitation's link. Now we only remove
it when the user completes registration.

Fixes: #12281
2020-05-26 21:45:44 -07:00
Tim Abbott 368e9572cc billing: Clean up copy on free trial pages.
This fixes some issues with unclear terminology and visual styling in
the pages for the new free trial.

There's probably more we can and should usefully do in the future.
2020-05-26 21:31:07 -07:00
Vishnu KS 8784539d53 free trial: Send users to /upgrade after realm creation. 2020-05-26 17:01:32 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
sahil839 1aebf3cab9 actions: Merge do_change_is_admin and do_change_is_guest.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.

do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
2020-05-25 16:17:10 -07:00
sahil839 9b78a73e36 populate_db: Add new admin user as 'Desdemona'.
This commit adds a second admin user named 'Desdemona' to dev and
test database.
2020-05-19 11:42:27 -07:00
Anders Kaseorg 78c70b1424 bugdown: Leave link titles alone until clean_user_content_links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Tim Abbott 200209995b test_signup: Update email content for last design tweak. 2020-05-08 15:18:21 -07:00
Tim Abbott a920544bc3
invite emails: Ensure user-controlled input is always in links.
Popular email clients like Gmail will automatically linkify link-like
content present in an HTML email they receive, even if it doesn't have
links in it.  This made it possible to include what in Gmail will be a
user-controlled link in invitation emails that Zulip sends, which a
spammer/phisher could try to take advantage of to send really bad spam
(the limitation of having the rest of the invitation email HTML there
makes it hard to do something compelling here).

We close this opportunity by structuring our emails to always show the
user's name inside an existing link, so that Gmail won't do new
linkification, and add a test to help ensure we don't remove this
structure in a future design change.

Co-authored-by: Anders Kaseorg <andersk@mit.edu>
2020-05-08 14:02:41 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
jagansivam28 bbf5a5efed invitation: Make Member to see invitations sent by him/her.
Member of the org can able see list of invitations sent by him/her.
given permission for the member to revoke and resend the invitations
sent by him/her and added tests for test member can revoke and resend
the invitations only sent by him/her.

Fixes #14007.
2020-05-01 17:42:49 -07:00
Mateusz Mandera f1ec02b40a auth: Add ExternalAuthResult to manage data in authentication flows.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).

The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.

This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.

Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
2020-04-28 22:19:02 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Udit107710 16218d6de3 streams: Remove dependency of streams on actions.
Refactored code in actions.py and streams.py to move stream related
functions into streams.py and remove the dependency on actions.py.

validate_sender_can_write_to_stream function in actions.py was renamed
to access_stream_for_send_message in streams.py.
2020-04-18 16:56:59 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Anders Kaseorg 4f748fb627 markdown: Stop setting target="_blank".
This setting is being overridden by the frontend since the last
commit, and the security model is clearer and more robust if we don't
make it appear as though the markdown processor is handling this
issue.

Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-01 14:01:45 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
arpit551 8f7733cb20 emails: Added placeholders strings in FormAddress.
We've had a bug for a while that if any ScheduledEmail objects get
created with the wrong email sender address, even after the sysadmin
corrects the problem, they'll still get errors because of the objects
stored with the wrong format.

We solve this by using FromAddress placeholders strings in
send_future_email function, so that ScheduledEmail objects end up
setting the final `from_address` value when mail is actually sent
using the setting in effect at that time.

Fixes #11008.
2020-03-27 16:41:02 -07:00
Steve Howell 1306239c16 tests: Use email/delivery_email more explicitly.
We try to use the correct variation of `email`
or `delivery_email`, even though in some
databases they are the same.

(To find the differences, I temporarily hacked
populate_db to use different values for email
and delivery_email, and reduced email visibility
in the zulip realm to admins only.)

In places where we want the "normal" realm
behavior of showing emails (and having `email`
be the same as `delivery_email`), we use
the new `reset_emails_in_zulip_realm` helper.

A couple random things:

    - I fixed any error messages that were leaking
      the wrong email

    - a test that claimed to rely on the order
      of emails no longer does (we sort user_ids
      instead)

    - we now use user_ids in some place where we used
      to use emails

    - for IRC mirrors I just punted and used
      `reset_emails_in_zulip_realm` in most places

    - for MIT-related tests, I didn't fix email
      vs. delivery_email unless it was obvious

I also explicitly reset the realm to a "normal"
realm for a couple tests that I frankly just didn't
have the energy to debug.  (Also, we do want some
coverage on the normal case, even though it is
"easier" for tests to pass if you mix up `email`
and `delivery_email`.)

In particular, I just reset data for the analytics
and corporate tests.
2020-03-19 16:04:03 -07:00
Steve Howell 1b16693526 tests: Limit email-based logins.
We now have this API...

If you really just need to log in
and not do anything with the actual
user:

    self.login('hamlet')

If you're gonna use the user in the
rest of the test:

    hamlet = self.example_user('hamlet')
    self.login_user(hamlet)

If you are specifically testing
email/password logins (used only in 4 places):

    self.login_by_email(email, password)

And for failures uses this (used twice):

    self.assert_login_failure(email)
2020-03-11 17:10:22 -07:00
Mateusz Mandera fe0f381914 populate_db: Don't restrict email domains by default in tests and dev.
The email domain restriction to @zulip.com is annoying in development
environment when trying to test sign up. For consistency, it's best to
have tests use the same default, and the tests that require domain
restriction can be adjusted to set that configuration up for themselves
explicitly.
2020-03-07 18:38:59 -08:00
Steve Howell 5e2a32c936 tests: Use users in send_*_message.
This commit mostly makes our tests less
noisy, since emails are no longer an important
detail of sending messages (they're not even
really used in the API).

It also sets us up to have more scrutiny
on delivery_email/email in the future
for things that actually matter.  (This is
a prep commit for something along those
lines, kind of hard to explain the full
plan.)
2020-03-07 18:30:13 -08:00
Steve Howell 7e55cab429 invite performance: Reduce queries to find existing users.
In the prep commits leading up to this, we split
out two new helpers:

    validate_email_is_valid
    get_errors_for_new_emails

Now when we validate invites we use two separate
loops to filter our emails.

Note that the two extracted functions map to two
of the data structures that used to be handled
in a single loop, and now we break them out:

    errors = validate_email_is_valid
    skipped = get_errors_for_new_emails

The first loop checks that emails are even valid
to begin with.

The second loop finds out whether emails are already
in use.

The second loop takes advantage of this helper:

    get_errors_for_new_emails

The second helper can query all potential new emails
with a single round trip to the database.

This reduces our query count.
2020-03-06 11:53:22 -08:00
Steve Howell 30b43605c3 invite performance: Reduce RealmDomain queries.
We now use the `get_realm_email_validator()`
helper to build an email validator outside
the loop of emails in our invite list.

This allows us to perform RealmDomain queries
only once per request, instead of once per
email.
2020-03-06 11:53:22 -08:00
Steve Howell c43a29ff54 invites: Fix bug with inviting cross realm bots.
Without the fix here, you will get an exception
similar to below if you try to invite one of the
cross realm bots.  (The actual exception is
a bit different due to some rebasing on my branch.)

	  File "/home/zulipdev/zulip/zerver/lib/request.py", line 368, in _wrapped_view_func
		return view_func(request, *args, **kwargs)
	  File "/home/zulipdev/zulip/zerver/views/invite.py", line 49, in invite_users_backend
		do_invite_users(user_profile, invitee_emails, streams, invite_as)
	  File "/home/zulipdev/zulip/zerver/lib/actions.py", line 5153, in do_invite_users
		email_error, email_skipped, deactivated = validate_email(user_profile, email)
	  File "/home/zulipdev/zulip/zerver/lib/actions.py", line 5069, in validate_email
		return None, (error.code), (error.params['deactivated'])
	TypeError: 'NoneType' object is not subscriptable

Obviously, you shouldn't try to invite a cross
realm bot to your realm, but we want a reasonable
error message.

RESOLUTION:

Populate the `code` parameter for `ValidationError`.

BACKGROUND:

Most callers to `validate_email_for_realm` simply catch
the `ValidationError` and then report a more generic error.

That's also what `do_invite_users` does, but it has the
somewhat convoluted codepath through `validate_email`
that triggers this code:

    try:
        validate_email_for_realm(user_profile.realm, email)
    except ValidationError as error:
        return None, (error.code), (error.params['deactivated'])

The way that we're using the `code` parameter for
`ValidationError` feels hacky to me.  The intention
behind `code` is to provide a descriptive error to
calling code, and it's not intended for humans, and
it feels strange that we actually translate this in
other places.  Here are the Django docs:

    https://docs.djangoproject.com/en/3.0/ref/forms/validation/

And then here's an example of us actually translating
a code (not part of this commit, just providing context):

    raise ValidationError(_('%s already has an account') %
                          (email,), code = _("Already has an account."),
                          params={'deactivated': False})

Those codes eventually get put into InvitationError, which
inherits from JsonableError, and we do actually display
these errors in the webapp:

    if skipped and len(skipped) == len(invitee_emails):
        # All e-mails were skipped, so we didn't actually invite anyone.
        raise InvitationError(_("We weren't able to invite anyone."),
                              skipped, sent_invitations=False)

I will try to untangle this somewhat in upcoming commits.
2020-03-06 11:53:22 -08:00
Steve Howell 923e6dcd5d tests: Add test for mirror_dummy user invites.
We allow folks to invite emails that are
associated with a mirror_dummy account.

We had a similar test already for registration,
but not invites.

This logic typically affects MIT realms in the
real world, but the logic should apply to any
realm, so I use accounts from the zulip realm
for convenient testing.  (For example, we might
run an IRC mirror for a non-MIT account.)
2020-03-06 11:53:22 -08:00
Steve Howell 332f8e7dde tests: Add query count check for invites.
I use a range here because there's some leak
from another test that causes the count to
vary.  Once we get this a bit more under control,
we should be able to analyze the leak better.
2020-03-06 11:53:22 -08:00
Steve Howell 32e1c22c88 tests: Improve test_invite_existing_user.
The substantive improvement here is to use
a strange casing for Hamlet's email, which
will prevent future casing bugs.

I also log in as Cordelia to prevent confusion
that the test has something to do with
inviting yourself.  It's more typical for
somebody to invite another person to a realm
(not realizing they're already there).

I also made two readability tweaks.
2020-03-06 11:53:22 -08:00
Mateusz Mandera efb3065158 social_auth: Take user to find_account if invalid subdomain is given.
This allows to also clean up some code that's not really useful.
2020-02-27 17:27:55 -08:00
Mateusz Mandera c78d0712f7 tests: For ldap tests, give each ldap user a unique password.
To avoid some hidden bugs in tests caused by every ldap user having the
same password, we give each user a different password, generated based
on their uids (to avoid some ugly hard-coding in a bunch of places).
2020-02-19 14:46:29 -08:00
Mateusz Mandera 6a0b68bc7f models: Delete get_stream_recipient function and its uses.
With recipient being now a Stream field, there's no more use for
this helper function.
2020-02-18 10:49:14 -08:00
Mateusz Mandera bde495db87 registration: Add support for mobile and desktop flows.
This makes it possible to create a Zulip account from the mobile or
desktop apps and have the end result be that the user is logged in on
their mobile device.

We may need small changes in the desktop and/or mobile apps to support
this.

Closes #10859.
2020-02-12 11:22:16 -08:00
Steve Howell f1ac16973c tests: Create signups stream in RealmCreationTests. 2020-02-10 15:45:13 -08:00
Mateusz Mandera 7b34853328 rate_limiter: Rename authenticate domain to authenticate_by_username.
This prepares for adding authenticate_by_ip_address.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 7c78d8a966 rate_limiter: Limit the amount of password reset emails to one address.
This limits the possibility to use the password reset form to make us
spam an email address with password reset emails.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 06198af5b9 auth: Handle rate limiting in OurAuthenticationForm and user_settings.
These parts of the code should catch the RateLimited exception and
generate their own, apprioprate user-facing error message.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 05e08891b2 auth: Replace deprecated password_reset_confirm.
Tests require adjusting, because the class-based view has an additional
redirect - through /uid/set-password/ and the token is read from the
session. See Django code of PasswordResetConfirmView.
2020-02-02 17:03:10 -08:00
Hashir Sarwar 13b3eb24b0 invitations: Avoid adding to notifications stream unconditionally.
Adding invited users to the notifications stream unconditionally isn't
a correct behaviour for guest users, where the previous behavior of
including the notifications stream no longer makes sense. Therefore,
while inviting a new user, the notifications stream is listed along
with other streams with a message "recieves notifications for new
streams" in order to distinguish it from other streams.

Fixes #13645.
2020-01-27 15:36:59 -08:00
Mateusz Mandera 01288ede9e recipients: Remove bulk_get_recipients function and its uses.
With the recipient field being denormalized into the UserProfile and
Streams models, all current uses of bulk_get_recipients can be done more
efficient, by simply checking the .recipient_id attribute on the
appropriate objects.
2019-12-12 12:00:13 -08:00
Mateusz Mandera 9995dab095 messages: Save a database query in check_message code path.
The flow in recipient_for_user_profiles previously worked by doing
validation on UserProfile objects (returning a list of IDs), and then
using that data to look up the appropriate Recipient objects.

For the case of sending a private message to another user, the new
UserProfile.recipient column lets us avoid the query to the Recipient
table if we move the step of reducing down to user IDs to only occur
in the Huddle code path.
2019-12-12 11:49:01 -08:00
Ryan Rehman 2589065405 tests: Rename invitor to inviter in test_signup and test_queue_worker.
"Inviter" seems to be preferred for the person who invites an invitee.
2019-12-10 17:22:32 -08:00
Mateusz Mandera 8acfa17fe6 models: Add recipient foreign key in UserProfile and Stream.
This is adds foreign keys to the corresponding Recipient object in the
UserProfile on Stream tables, a denormalization intended to improve
performance as this is a common query.

In the migration for setting the field correctly for existing users,
we do a direct SQL query (because Django 1.11 doesn't provide any good
method for doing it properly in bulk using the ORM.).

A consequence of this change to the model is that a bit of code needs
to be added to the functions responsible for creating new users (to
set the field after the Recipient object gets created).  Fortunately,
there's only a few code paths for doing that.

Also an adjustment is needed in the import system - this introduces a
circular relation between Recipient and UserProfile. The field cannot be
set until the Recipient objects have been created, but UserProfiles need
to be created before their corresponding Recipients. We deal with this
by first importing UserProfiles same way as before, but we leave the
personal_recipient field uninitialized. After creating the Recipient
objects, we call a function to set the field for all the imported users
in bulk.

A similar change is made for managing Stream objects.
2019-12-09 15:14:41 -08:00
Mateusz Mandera fcc91ae370 ldap: Disallow creating non-ldap accounts matching LDAP_APPEND_DOMAIN.
In configurations with LDAP_APPEND_DOMAIN, we don't want people creating
non-ldap accounts with emails matching the ldap domain.
So in the registration flow, if the email isn't found in LDAP, but
matches LDAP_APPEND_DOMAIN, we stop, rather than proceeding with account
creation. In case of emails not matching LDAP_APPEND_DOMAIN, we will
still continue to make a normal, non-ldap account.
2019-12-02 17:44:11 -08:00
Mateusz Mandera 06c2161f7e auth: Use zxcvbn to ensure password strength on server side.
For a long time, we've been only doing the zxcvbn password strength
checks on the browser, which is helpful, but means users could through
hackery (or a bug in the frontend validation code) manage to set a
too-weak password.  We fix this by running our password strength
validation on the backend as well, using python-zxcvbn.

In theory, a bug in python-zxcvbn could result in it producing a
different opinion than the frontend version; if so, it'd be a pretty
bad bug in the library, and hopefully we'd hear about it from users,
report upstream, and get it fixed that way. Alternatively, we can
switch to shelling out to node like we do for KaTeX.

Fixes #6880.
2019-11-21 10:23:37 -08:00
Mateusz Mandera ed40d37e44 ldap: Fix realm_creation=True registration flow.
When creating realm with the ldap backend, the registration flow didn't
properly handle some things - the user wouldn't be set as realm admin,
initial subscriptions and messages weren't created, and the redirect
wasn't happening properly in the case of subdomains.
2019-11-08 14:01:45 -08:00
Mateusz Mandera 071038f111 test_signup: Check that user is admin in RealmCreationTest. 2019-11-08 14:01:45 -08:00
Mateusz Mandera 5aded51b73 register: Pre-populate Name in social backend flow.
By adding some additional plumbing (through PreregistrationUser) of the
full_name and an additional full_name_validated option, we
pre-populate the Full Name field in the registration form when coming
through a social backend (google/github/saml/etc.) and potentially skip
the registration form (if the user would have nothing to do there other
than clicking the Confirm button) and just create the account and log
the user in.
2019-11-03 16:15:48 -08:00
Mateusz Mandera 339f742578 test_signup: Add test for a non-covered case in accounts_register. 2019-11-03 15:51:19 -08:00
Mateusz Mandera 8c065d1fcd ldap: Ensure django_to_ldap_username returns username that is in ldap.
This changes the way django_to_ldap_username works to make sure the ldap
username it returns actually has a corresponding ldap entry and raise an
exception if that's not possible. It seems to be a more sound approach
than just having it return its best guess - which was the case so far.
Now there is a guarantee that what it returns is the username of an
actual ldap user.

This allows communicating to the registration flow when the email being
registered doesn't belong to ldap, which then will proceed to register
it via the normal email backend flow - finally fixing the bug where you
couldn't register a non-ldap email even with the email backend enabled.

These changes to the behavior of django_to_ldap_username require small
refactorings in a couple of other functions that call it, as well as
adapting some tests to these changes. Finally, additional tests are
added for the above-mentioned registration flow behavior and some
related corner-cases.
2019-10-25 12:14:51 -07:00
Mateusz Mandera 1be2779515 tests: Add ldap_username() and ldap_password() method. 2019-10-22 16:03:58 -07:00
Vishnu KS e080b42fe5 tests: Move get_stream_id function to ZulipTestCase. 2019-10-21 16:56:48 -07:00
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Mateusz Mandera 72dd834088 test_signup: Migrate ldap tests to the new format. 2019-10-17 17:10:39 -07:00
Tim Abbott d364891894 ldap: Fix password prompt when configured only to populate data.
Previously, the logic for determining whether to provide an LDAP
password prompt on the registration page was incorrectly including it
if any LDAP authentication was backend enabled, even if LDAP was
configured with the populate-only backend that is not responsible for
authentication (just for filling in name and custom profile fields).

We fix this by correcting the conditional, and add a test.

There's still follow-up work to do here: We may still end up
presenting a registration form in situations where it's useless
because we got all the data from SAML + LDAP.  But that's for a future
issue.

This fixes a bug reported in #13275.
2019-10-17 14:46:46 -07:00
Rishi Gupta 01bf99ec65 billing: Keep track of role count changes in RealmAuditLog. 2019-10-08 17:18:30 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Tim Abbott b8bfdebea1 onboarding: Mark the most recent 20 messages as unread.
After a new user joins an active organization, it isn't obvious what
to do next; this change causes there to be recent unread messages in
the stream sidebar for the user to click on to get a feel for what's
happening in the organization and experiment with Zulip.

Fixes #6512.
2019-08-26 13:51:52 -07:00
Rishi Gupta e058558a52 emails: Send invitation reminder email two days before expiry.
Hopefully this does a better job of spurring people to action, and also
suggests a self-service fix if they don't (i.e. contacting the person that
invited them).
2019-08-23 12:53:11 -07:00
Tim Abbott 7844f87d65 actions: Tag already has an account string for translation.
And also simplify an another validation string.

Fixed #9473.
2019-08-21 21:16:47 -07:00
Mateusz Mandera 26f7dd09ce display_recipient: Eliminate raw_display_recipient from message dicts.
This was used as a helper to construct the final display_recipient when
fetching messages. With the new mechanism of constructing
display_recipient by fetching appropriate users/streams from the
database and cache, this shouldn't be needed anymore.
2019-08-20 12:22:52 -07:00
Mateusz Mandera c779bb1959 messages: Don't use display_recipient values from cached message_dicts.
The user information in display_recipient in cached message_dicts
becomes outdated if the information is changed in any way.

In particular, since we don't have a way to find all the message
objects that might contain PMs after an organization toggles the
setting to hide user email addresses from other users, we had a
situation where client might see inaccurate cached data from before
the transition for a period of up to hours.

We address this by using our generic_bulk_cached_fetch toolchain to
ensure we always are fetching display_recipient data from the database
(and/or a special recipient_id -> display_recipient cache, which we
can flush easily).

Fixes #12818.
2019-08-20 12:15:30 -07:00
Mateusz Mandera d3d62f99e4 tests: Adjust to new changes in populate_db. 2019-07-31 16:01:37 -07:00
Yashashvi Dave 8e269b4651 models: Rename notification to `enable_stream_audible_notifications`.
Rename notification property `enable_stream_sounds` to
`enable_stream_audible_notifications` to match with other
notification property patterns.

Fixes part of #12304
2019-06-12 16:24:51 -07:00
Mayank Madan 7fedcbd840 tests: Extract and use assert_logged_in_user_id test helper.
This cleans up the pattern for how we check which user is logged in
during Zulip's backend unit tests to be much more readable (replacing
the arcane session code that does this check).
2019-05-27 18:32:27 -07:00
Mayank Madan d5e61e20e5 dev tooling: Add "create new user" and "create new realm" buttons.
Significantly tweaked by tabbott to clean up and expand the tests.

Fixes: #6018.
2019-05-21 15:03:37 -07:00
Anders Kaseorg 9efda71a4b get_realm: raise DoesNotExist instead of returning None.
This makes the implementation of `get_realm` consistent with its
declared return type of `Realm` rather than `Optional[Realm]`.

Fixes #12263.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-06 21:58:16 -07:00
Rishi Gupta 6c2c351894 onboarding: Update initial messages and stream descriptions. 2019-03-21 12:34:24 -07:00
Rishi Gupta e8741c448d refactoring: Move set_default_streams into do_create_realm.
After the commits leading up to this, the only meaningful use of this
function was in the realm creation process.
2019-03-21 12:33:19 -07:00
Rishi Gupta e71a1a2b4e onboarding: Remove initial streams other than general and core team.
The hope is that by having a shorter list of initial streams, it'll
avoid some potential confusion confusion about the value of topics.
At the very least, having 5 streams each with 1 topic was not a good
way to introduce Zulip.

This commit minimizes changes to the message content in
`send_initial_realm_messages` to keep the diff readable. Future commits will
reshape the content.
2019-03-21 12:30:14 -07:00
Raymond Akornor d8a2bb2b7c deliver_email: Refactor to extract email delivery.
This is a follow up on #11181. We extract email delivery into
`zerver/lib/send-email.py` to make it cleaner and easy to test.
2019-03-20 14:52:05 -07:00
Hemanth V. Alluri 1269b72f47 context_processors: Do not render inline previews for realm description.
We never intended to render them for this use case as the result would
not look good, and now we have a convenient bugdown option for
controlling this behavior.

Since we're not storing the markdown rendering anywhere, there's
conveniently no data migration required.

Fixes #11889.
2019-03-17 13:55:10 -07:00
Tim Abbott 50dc317466 notifications: Rename notifications.py to email_notifications.py.
This library is entirely about email notifications specifically, and
this rename should help make the codebase more readable.
2019-03-15 11:02:17 -07:00
Raymond Akornor 89351cdd19 send_email: Add ScheduledEmail support for multiple recipients.
Follow up on 92dc363. This modifies the ScheduledEmail model
and send_future_email to properly support multiple recipients.

Tweaked by tabbott to add some useful explanatory comments and fix
issues with the migration.
2019-03-15 11:02:12 -07:00
Tim Abbott 983e24a7f5 auth: Use HTTP status 404 for invalid realms.
Apparently, our invalid realm error page had HTTP status 200, which
could be confusing and in particular broken our mobile app's error
handling for this case.
2019-03-14 13:50:09 -07:00
Harshit Bansal 216b7b0a19 auth: Remove `invalid_subdomain` restriction from LDAP backend.
Fixes: #11692.
2019-03-04 11:53:53 -08:00
Vishnu Ks 763eca6ca9 invites: Add UI for revoking multiuse invites. 2019-02-15 14:13:31 -08:00
Vishnu Ks 410e2574d1 settings: Remove unused setting ACCOUNT_ACTIVATION_DAYS.
ACCOUNT_ACTIVATION_DAYS doesn't seems to be used anywhere.
INVITATION_LINK_VALIDITY_DAYS seems to do it's job currently.

(It was only ever used in very early Zulip commits).
2019-02-15 13:56:10 -08:00
Vishnu Ks 868a763cec auth2: Don't use session for passing multiuse invite key.
For Google auth, the multiuse invite key should be stored in the
csrf_state sent to google along with other values like is_signup,
mobile_flow_otp.

For social auth, the multiuse invite key should be passed as params to
the social-auth backend. The passing of the key is handled by
social_auth pipeline and made available to us when the auth is
completed.
2019-02-12 15:51:11 -08:00
Harshit Bansal 1a5e07e0f9 ldap: Add ability to automatically sync custom profile fields. 2019-02-05 10:25:50 -08:00
Anders Kaseorg 3127fb4dbd zerver/tests: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:43:03 -08:00
Shubham Dhama 5779320874 invites: Use stream_id instead of stream names in invitation.
This replaces the current usage of stream names with stream ids.

This commit also removes the `traditional` attribute from the invite
form as now we are sending stream_ids as an argument; this was the
only place in the codebase we used traditional=true, and it's great to
have it removed.
2019-02-01 15:47:20 -08:00
Rishi Gupta 7c11fe819a billing: Update license ledger when users are added and removed. 2019-01-28 16:04:11 -08:00
Harshit Bansal fcf2ffe8db registration: Populate LDAP users using invitation information.
Fixes: #11212.
2019-01-17 10:16:48 -08:00
Harshit Bansal 3e3d8527b0 tests: Use `init_fakeldap()` in `test_signup.py`. 2019-01-17 10:16:48 -08:00
Harshit Bansal 47b5f9a4a3 ldap: Don't ask users to create password when invited. 2019-01-17 10:16:48 -08:00
Harshit Bansal 356c5bfb0e dev_ldap: Make `userPassword` a multi-value attribute.
`fakeldap` assumes every attribute to be a multi-value attribute
while making comparison in `_comapare_s()` and so while making
comparisons for password it gives a false positive. The result
of this was that it was possible to login in the dev environment
using LDAP using a substring of the password. For example, if the
LDAP password is `ldapuser1` even entering `u` would log you in.
2019-01-17 10:16:48 -08:00
Harshit Bansal 2dc500b11e tests: Fix tests broken in previous commit and add some new ones.
This commit fixes some tests broken in commit
32aa4e02ca.
It also adds some tests for full name mapping in the registration
flow using LDAP.
2019-01-16 09:45:51 -08:00
Vaibhav 4219cc497d tooling: Move confirmation_key view to development only views.
Previously, zerver.views.registration.confirmation_key was only
available in development; now we make that more structurally clear by
moving it to the special zerver/views/development directory.

Fixes #11256.
2019-01-11 12:45:21 -08:00
Shubham Dhama efb9128aaa invite: Add option to invite user as guest.
This completes our basic guest user feature.

Fixes: #10818.
2019-01-05 14:52:51 -08:00
Shubham Dhama 42c262b807 invite: Replace `invite_as_admin` usage with `invite_as`.
Since we have already added the `invite_as` field to models, we can now
replace usage of `invite_as_admin` properly with its equivalent `invite_as
== PreregistrationUser.INVITE_AS['REALM_ADMIN']`.

Hence, also removed now redundant `invite_as`.
2019-01-05 14:46:38 -08:00
Vishnu Ks e5f7d65231 signup: Set oldest account as default option in import settings.
Fixes: #11018
2018-12-29 15:01:09 -08:00
Rishi Gupta 0844c9a6ec emails: Update text for password_reset.
Note that a pretty common use case for this is a realm admin sending this to
everyone after an import from HipChat or Slack. So this adds the realm_name
to the title (so that there is something they might recognize) and kept the
wording generic enough to accommodate the user not having clicked anything
to get this email.

Also strengthens the tests a bit to better test the complicated template
logic.
2018-12-20 16:26:19 -08:00
ishanrai05 4105fb683b notifications: Optimize push notifications code path in tests.
This checks if push_notification_enabled() is set to false in
handle_push_notification and adds an early return statement.

This is a significant performance optimization for our unit tests
because the push notifications code path does a number of database
queries, and this migration means we don't end up doing those queries
the hundreds of times we send PMs or mentions in our tests where we're
not trying to test the push notifications functionality.

This should also have a small message sending scalability improvement
for any Zulip servers without push notifications enabled.

Tweaked by tabbott to fix a few small issues.

Fixes #10895.
2018-12-15 11:12:43 -08:00
Tim Abbott ee901ac8b1 django: Remove the Django contrib Sites app to fix 2FA QR codes.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).

We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.

Fixes #11014.
2018-12-12 15:01:55 -08:00
Max Nussenbaum 536de666df register: Style avatar that shows when importing settings.
This styles the avatar and username that show when the registering
user is importing their settings from an existing Zulip account.

Tweaked by tabbott to fix the test/linter failures, a bit of styling,
and tag strings for translation.
2018-12-10 15:40:30 -08:00
Vishnu Ks 18f90754da portico: Change wording of finding organization in /accounts/go. 2018-12-06 09:24:22 -08:00
Vishnu Ks fc4b6d8f5e auth: Redirect login to realm_redirect page instead of find_accounts. 2018-12-04 09:35:35 -08:00
Vishnu Ks 788b98d041 portico: Add page for redirecting to a realm subdomain. 2018-12-04 09:35:35 -08:00
Raymond Akornor 92dc3637df send_email: Add support for multiple recipients.
This adds a function that sends provided email to all administrators
of a realm, but in a single email. As a result, send_email now takes
arguments to_user_ids and to_emails instead of to_user_id and
to_email.

We adjust other APIs to match, but note that send_future_email does
not yet support the multiple recipients model for good reasons.

Tweaked by tabbott to modify `manage.py deliver_email` to handle
backwards-compatibily for any ScheduledEmail objects already in the
database.

Fixes #10896.
2018-12-03 15:12:11 -08:00
Rishi Gupta dd0126ff1b settings: Improve error message when deactivating the last user.
This PR was originally started by Rishi Gupta (see #10383).
2018-11-27 12:49:43 -08:00
Vishnu Ks 3258ce1fa4 emails: Cleanup followup day1 email. 2018-11-19 11:16:36 -08:00
Roger Souza 6179bba2bd emails: Add one-click unsubscribe link to new login emails.
Fixes: #10547.
2018-11-08 14:39:33 -08:00
Tim Abbott 17be6e9165 confirmation: Move one_click_unsubscribe_link out of notifications.py.
This helps prevent a problematic import sequence when we start using
it from signals.py.
2018-11-08 14:39:33 -08:00
Rishi Gupta 7acb4e9165 portico: Update text for reset_emailed.html.
The current text may have made sense with different styling, but it feels
harsh now.
2018-09-15 18:45:10 -07:00
Harshit Bansal e7311cdf5d emoji: Bring back the feature of changing emojisets.
This is largely inspired by requests from people not liking the
Google's new emojiset. A lot of people were requesting to revert
back to old blobs emojiset so we are re-enabling this feature
after making relevant infrastructure changes for supporting google's
old blob emojiset and re-adding support for twitter emojiset.

Fixes: #10158.
2018-08-31 11:52:24 -07:00
Vishnu Ks 62b93d551a signup: Fix broken resend email link in realm creation flow.
Also use name for selecting form in casper tests
as form with action=new is present in both /new
and /accounts/new/send_confirm/ which breaks
test in CircleCI as
waitWhileVisible('form[action^="/new/"]) never stops
waiting.
2018-08-26 22:54:05 -07:00
Vishnu Ks d2e4417a72 urls: Separate endpoint for signup and new realm email confirm.
This is preparation for the next commit.
2018-08-26 22:53:57 -07:00
Shubham Dhama e70cf3bd67 emails/password_reset: Change text for listing other active accounts.
The main benefit of this change is that it reduces the amount of total
string that we might need to tag for translation.

Fixes: #10323.
2018-08-22 17:49:40 -07:00
Shubham Dhama aa8b3d2beb emails/password_reset: Change template string for deactivated users. 2018-08-22 17:43:23 -07:00
Steve Howell cac9a02d6d tests: Add coverage for check_invite_limit().
The function being tested here was kind of an
emergency response to some spam attacks.  It
works for a pretty specific set of circumstances,
so it requires a lot of setup.

We may eliminate this function as we improve
our realm "plan types", and if that happens, we
can either eliminate this test or repurpose it.
2018-08-22 16:52:30 -07:00
Steve Howell 948190c9ff tests: Test spam prevention feature for invites.
We have code to prevent newbies on open realms
from inviting users.  This is mostly intended
to hinder spammers.  This commit just adds some
test coverage.
2018-08-17 08:14:42 -07:00
Rishi Gupta 4813bac98b signup: Add opayq.com to disposable domain whitelist. 2018-08-13 10:56:47 -07:00
Rishi Gupta 228e237d51 tests: Fix incorrect comment in test_invite_using_disposable_email. 2018-08-13 10:56:47 -07:00
Tim Abbott da8f4bc0e9 push notifications: Add support for removing GCM push notifications.
This uses the recently introduced active_mobile_push_notification
flag; messages that have had a mobile push notification sent will have
a removal push notification sent as soon as they are marked as read.

Note that this feature is behind a setting,
SEND_REMOVE_PUSH_NOTIFICATIONS, since the notification format is not
supported by the mobile apps yet, and we want to give a grace period
before we start sending notifications that appear as (null) to
clients.  But the tracking logic to maintain the set of message IDs
with an active push notification runs unconditionally.

This is designed with at-least-once semantics; so mobile clients need
to handle the possibility that they receive duplicat requests to
remove a push notification.

We reuse the existing missedmessage_mobile_notifications queue
processor for the work, to avoid materially impacting the latency of
marking messages as read.

Fixes #7459, though we'll need to open a follow-up issue for
using these data on iOS.
2018-08-10 13:58:39 -07:00
Tim Abbott 58307f80aa event_queue: Stop mocking push notifications in most tests.
Historically, queue_json_publish had a special third argument that was
basically its default mock behavior in the test suite.  We've been
migrating away from that model, because it was confusing and resulted
in poor test coverage of our queue worker code paths; this was one of
the last holdouts.

As it turns out, we don't exercise this code path in a way that
impacts tests much; the main downside of this change is a likely small
penalty to performance of the full test suite when sending private
messages.
2018-08-10 13:58:39 -07:00
Tim Abbott 83300c329b register: Fix period at end of heading. 2018-08-09 17:17:45 -07:00
Yago González 298aa0fcbf mobile: Make otp_encrypt_api_key accept API keys.
Since otp_encrypt_api_key only encrypts API keys, it doesn't require
access to the full UserProfile object to work properly. Now the
parameter it accepts is just the API key.

This is preparatory refactoring for removing the api_key field on
UserProfile.
2018-08-08 16:45:40 -07:00
Roman Godov c0806917ec models: Rename Realm.restricted_to_domain field.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-07-31 09:28:33 -07:00
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Vishnu Ks 1b179ca530 signup: Prevent users from signing up with email containing +. 2018-06-23 12:03:30 -07:00
Vishnu Ks e34fcf982f registration: Use tokenized noreply address in user invite. 2018-06-23 12:03:30 -07:00
Vishnu Ks 994e1a2154 registration: Use tokenized noreply address in password reset. 2018-06-23 12:03:30 -07:00
Vishnu Ks 74351fb250 send_email: Add function to generate tokenized noreply in FromAddress. 2018-06-23 12:03:30 -07:00
Vishnu Ks 4da61e2a17 signup: Copy enter_sends value when importing settings. 2018-06-16 08:50:49 -07:00
Vishnu Ks b639778d0f signup: Copy hotspots when importing settings. 2018-06-16 08:50:49 -07:00
Shubham Dhama 596ce5b60f invite: Make inviting new users inaccessible for guest users. 2018-06-16 06:33:13 -07:00
Vishnu Ks 53237d39aa registration: Allow users to import profile picture. 2018-06-06 13:29:42 -07:00
Tim Abbott 91ec0aba09 auth: Improve interactions between LDAPAuthBackend and EmailAuthBackend.
Previously, if you had LDAPAuthBackend enabled, we basically blocked
any other auth backends from working at all, by requiring the user's
login flow include verifying the user's LDAP password.

We still want to enforce that in the case that the account email
matches LDAP_APPEND_DOMAIN, but there's a reasonable corner case:
Having effectively guest users from outside the LDAP domain.

We don't want to allow creating a Zulip-level password for a user
inside the LDAP domain, so we still verify the LDAP password in that
flow, but if the email is allowed to register (due to invite or
whatever) but is outside the LDAP domain for the organization, we
allow it to create an account and set a password.

For the moment, this solution only covers EmailAuthBackend.  It's
likely that just extending the list of other backends we check for in
the new conditional on `email_auth_backend` would be correct, but we
haven't done any testing for those cases, and with auth code paths,
it's better to disallow than allow untested code paths.

Fixes #9422.
2018-05-28 22:47:47 -07:00
Tim Abbott 3842404cc0 ldap: Don't allow password reset for users in LDAP domain.
This is the analog of the last commit, for the password reset flow.
For these users, they should be managing/changing their password in
the LDAP server.

The error message for users doing the wrong thing here is nonexistent
isn't great, but it should be a rare situation.
2018-05-28 22:47:47 -07:00
Umair Khan e063bb0d50 2FA: Add tests for two-factor auth. 2018-05-23 15:49:35 -07:00
Vishnu Ks 8369e2b15e registration: Allow users to import settings from other realm.
This should significantly improve the user experience for creating
additional accounts on zulipchat.com.

Currently, disabled in production pending some work on visual styling.
2018-05-23 10:30:00 -07:00
Umair Khan 18defcc673 2FA: Fix check for context_data attr.
This attribute only makes sense for responses that are generated using a
template.
2018-05-21 07:50:05 -07:00
Tim Abbott 336ad0fbb1 password reset: Handle deactivated users and realms.
Since this is a logged-out view, need to actually write code for the
case of deactivated realms.

The change to get_active_user is more for clarity; the Django password
reset form already checks for whether the user is active earlier.
2018-05-20 20:02:27 -07:00
Steve Howell 4332fd64f7 Add submessages to message payloads. 2018-05-16 15:13:33 -07:00
Aditya Bansal 5416d137d3 zerver/tests: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Vishnu Ks 733da0ac07 settings: Fix 500 when trying to change email to disposable email.
Fixes #9240
2018-05-04 10:52:38 -07:00
Preston Hansen 0258d7db0d slack import: Be less strict in `check_subdomain_available`.
If the sysadmin is doing something explicit in a management command,
it's OK to take a reserved or short subdomain.

Fixes #9166.
2018-04-23 11:48:12 -07:00
Tim Abbott c65a4e8f0b test_signup: Remove unnecessary LoginOrAskForRegistrationTestCase.
This test class is basically a poor version of the end-to-end tests
that we have in `test_auth_backends.py`, and didn't really add any
value other than making it difficult to refactor.
2018-04-22 19:41:18 -07:00
Vishnu Ks e92838a31f registration: Catch email validation error and show error message. 2018-04-06 15:18:32 -07:00
Shubham Dhama b7aae89029 org settings: Add realm level default time format setting. 2018-03-31 16:23:26 -07:00
Aditya Bansal 1e48dac8f3 auth.py: Make redirects to 'next' url work for google and github.
In this commit we start to support redirects to urls supplied as a
'next' param for the following two backends:
* GoogleOAuth2 based backend.
* GitHubAuthBackend.
2018-03-21 13:35:44 -07:00
Nikhil Kumar Mishra a5472ddee7 initial_password: Add unit test for INITIAL_PASSWORD_SALT = None. 2018-03-19 10:36:26 -07:00