Commit Graph

167 Commits

Author SHA1 Message Date
Vishnu KS 1e68525f83 support: Ensure that only one form is posted at a time.
The forms to change plan_type, add discount, scrub_realm etc
all post to the same endpoint.

Our frontend code is written so that only one form posts at a time.
But there should be no harm in enforcing the same in backend as well.
2020-07-01 16:45:38 -07:00
Vishnu KS 4c6350fa4b billing: Add option to request a sponsorship in /upgrade. 2020-07-01 16:45:38 -07:00
sahil839 1f8f227444 models: Update values of PreregistrationUser.invite_as dict.
This commit changes the PreregistrationUser.invite_as dict to have
same set of values as we have for UserProfile.role.

This also adds a data migration to update the already exisiting
PreregistrationUser and MultiuseInvite objects.
2020-06-24 11:09:07 -07:00
wowol 035d047dff urls: Migrate analytics urls to use modern django patterns. 2020-06-23 15:02:42 -07:00
Anders Kaseorg 1a3441dbf5 confirmation: Pass realm rather than host to confirmation_url.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
arpit551 aa70baba71 analytics: Add backend for messages read over time graph.
This commit includes changes in the api /json/analytics/chart_data to
send data for the newly added graph, as well as tests.
2020-06-14 21:19:24 -07:00
arpit551 c4b5d09283 analytics: Add LoggingCount for messages read stats.
Whenever we use API queries to mark messages as read we now increment
two new LoggingCount stats, messages_read::hour and
messages_read_interactions::hour.

We add an early return in do_increment_logging_stat function if there
are no changes (increment is 0), as an optimization to avoid
unnecessary database queries.

We also log messages_read_interactions::hour Logging stat
as the number of API queries to mark messages as read.

We don't include tests for the case where do_update_pointer is called
because do_update_pointer will most likely be removed from the
codebase in the near future.
2020-06-14 21:15:27 -07:00
arpit551 27daf38587 analytics: Derive AnalyticsTestCase class from ZulipTestCase.
The ZulipTestCase class contains various useful utility functions that
we could be using in test_counts.
2020-06-14 21:08:24 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 5839fdf963 analytics: Improve escaping correctness with psycopg2.sql.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
arpit551 fb2aae1c02 analytics tests: Save recipient in stream object.
At the time of creating streams in test_counts.py we earlier did not saved
recipient in the stream object.

stream.recipient is used in many functions so they would throw error.

The right long-term fix here is probably to just use the standard
stream creation functions rather than having a hacky duplicate
here.
2020-06-08 11:33:24 -07:00
Anders Kaseorg 1f565a9f41 timezone: Use standard library datetime.timezone.utc consistently.
datetime.timezone is available in Python ≥ 3.2.  This also lets us
remove a pytz dependency from the PostgreSQL scripts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
Sahil Batra 77d4be56a4
users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
sahil839 9b78a73e36 populate_db: Add new admin user as 'Desdemona'.
This commit adds a second admin user named 'Desdemona' to dev and
test database.
2020-05-19 11:42:27 -07:00
sahil839 36dca5ba5c analytics: Add order_by to query used for fetching admin emails.
This commit adds order_by to the query that fetches admin emails
in views.py. This is added to show emails alphabetically in case
of multiple admins.
2020-05-19 11:42:27 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Vishnu KS 905f3d2d23 stats: Don't show analytics data unavailable error for new realms.
The value of start would be set to realm creation date or installation
date unless a value is explicitly provided by the user. We don't
want to show analytics data missing error in both these cases if
start is less than 24.5 hours from timezone_now() since some of the
stats are populated by a daily cron job that may take a couple of
minutes to run.

The same case applies if the value of the start is passed in the
request.
2020-04-03 11:51:20 -07:00
Steve Howell 1306239c16 tests: Use email/delivery_email more explicitly.
We try to use the correct variation of `email`
or `delivery_email`, even though in some
databases they are the same.

(To find the differences, I temporarily hacked
populate_db to use different values for email
and delivery_email, and reduced email visibility
in the zulip realm to admins only.)

In places where we want the "normal" realm
behavior of showing emails (and having `email`
be the same as `delivery_email`), we use
the new `reset_emails_in_zulip_realm` helper.

A couple random things:

    - I fixed any error messages that were leaking
      the wrong email

    - a test that claimed to rely on the order
      of emails no longer does (we sort user_ids
      instead)

    - we now use user_ids in some place where we used
      to use emails

    - for IRC mirrors I just punted and used
      `reset_emails_in_zulip_realm` in most places

    - for MIT-related tests, I didn't fix email
      vs. delivery_email unless it was obvious

I also explicitly reset the realm to a "normal"
realm for a couple tests that I frankly just didn't
have the energy to debug.  (Also, we do want some
coverage on the normal case, even though it is
"easier" for tests to pass if you mix up `email`
and `delivery_email`.)

In particular, I just reset data for the analytics
and corporate tests.
2020-03-19 16:04:03 -07:00
Mateusz Mandera b4ce167a88 models: Add recipient foreign key to Huddle.
This follows the already tested approach from
8acfa17fe6.
2020-03-17 05:41:11 -07:00
Steve Howell 1b16693526 tests: Limit email-based logins.
We now have this API...

If you really just need to log in
and not do anything with the actual
user:

    self.login('hamlet')

If you're gonna use the user in the
rest of the test:

    hamlet = self.example_user('hamlet')
    self.login_user(hamlet)

If you are specifically testing
email/password logins (used only in 4 places):

    self.login_by_email(email, password)

And for failures uses this (used twice):

    self.assert_login_failure(email)
2020-03-11 17:10:22 -07:00
arpit551 f299f31340 analytics: Fix missing unique constraint when subgroup is null.
Replaced unique_together with UniqueConstraint in models that
covered nullable fields as in unique_together database indexes
don't work where subgroup=None. So added conditional unique
index handling invalid duplicate Count data.

Added 0015_clear_duplicate_counts migration to handle existing
data that violates the constraints.

Also corrected a test case in test_counts.py which didn't clear its
state properly and thus was accidentally taking advantage of this
database schema bug.
2020-03-06 11:10:04 -08:00
arpit551 b23a5431cd analytics: Add realm argument to analytics.
This changeset is prepartory work for doing something reasonable with
analytics data during the zulip -> zulip data import process (and
potentially e.g. slack -> Zulip as well).

To support that, we need to make it possible to do our analytics
calculations for a single realm.

We do this while maintaining backwards compatibility and avoiding
massive duplicated code by adding an optional `realm` argument to the
entrypoints to the analytics system, especially process_count_stat.

More work involving restructuring FillState will be required for this
to be actually usable for its intented purpose, but this commit is a
nice checkpoint along the way.

Tweaked by tabbott to adjust comments and disable InstallationCount
updates when a realm argument is specified.
2020-01-23 17:36:13 -08:00
Vishnu Ks a26b379a14 support: Send confirmation email on realm activation. 2019-12-02 09:51:45 -08:00
Vishnu KS ec955f8f78 support: Show confirmation links in search.
Fixes #13060 #12784
2019-10-21 16:56:50 -07:00
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Tim Abbott a352f2e10d test_counts: Remove custom user creation code.
Like the last commit, this avoids us needing to update this random
test class in the analytics subsystem when we adjust the UserProfile
model.
2019-09-19 14:31:58 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Mateusz Mandera b321dcd50b test_views: Prepare for moving system bots to zulipinternal. 2019-07-24 16:26:10 -07:00
Vishnu Ks 14e582fb59 support: Add functionality to copy admin emails.
Also renamed a bunch of functions in test_views for better
readability.
2019-06-14 10:19:50 -07:00
Tim Abbott 1d44fd724b audit log: Log which server admin deactivated a realm too. 2019-05-08 15:09:48 -07:00
Rishi Gupta 0218da64f1 support: Update success message for plan type change. 2019-05-08 15:09:48 -07:00
Rishi Gupta 98da11c558 support: Rename deactive to deactivated. 2019-05-08 15:09:48 -07:00
Vishnu Ks 6c58603eaf support: Add support for scrubbing realm. 2019-05-06 20:12:54 -07:00
Vishnu Ks f6203f068b support: Add support for activating and deactivating realm. 2019-05-06 20:12:48 -07:00
Vishnu Ks 8eeb8280b4 activity: Create interface for doing support operations.
This should grow into a tool that makes it much easier to do common
organization management tasks without using a manage.py shell.
2019-03-11 12:01:11 -07:00
Rishi Gupta 9962377018 analytics: Fix midnight-related bug in test.
Previously, this would flake if the day changed between
user2 = do_create_user('email2', 'password', ...) and
do_deactivate_user(user2).
2019-02-28 14:48:30 -08:00
Anders Kaseorg f5197518a9 analytics/zilencer/zproject: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:31:45 -08:00
Rishi Gupta 85f7ac8172 analytics: Remove Anomaly model. 2019-02-01 18:48:18 -08:00
Tim Abbott 7ddcbd0d3a analytics: Set delivery_email in hacky test user creation code. 2018-12-06 15:33:28 -08:00
Steve Howell c0fd8660d2 subject -> topic: Fix analytics test (minor). 2018-11-14 23:24:06 -08:00
Yashashvi Dave 02a5849d4c statistics: Guest user can't access realm statistics.
Don't allow guest user to access realm statistics from
UI or at API level.

Fixes part of #10749.
2018-11-02 11:43:09 -07:00
Vishnu Ks 201b99a6f8 models: Add USER_REACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30