Commit Graph

428 Commits

Author SHA1 Message Date
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Vishnu KS 67bacd6e31 billing: Don't allow guest users to upgrade. 2020-07-22 16:57:49 -07:00
Vishnu KS cb01a7f599 billing: Restrict access to billing page to realm owners and billing admins. 2020-07-22 16:57:49 -07:00
Anders Kaseorg f2e7076e2a decorator: Replace type: ignore with cast, avoid Any.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg e582bbea4a decorator: Strengthen types of signature-preserving decorators.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 21b9ee2047 decorator: Fix bogus function=None case of zulip_login_required.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg dcbc8e66fb decorator: Remove authenticated_json_post_view.
It’s effectively a combination of require_post with
authenticated_json_view and has one use.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg ca1d9603cb decorator: Fix type of signature-changing decorators.
In a decorator annotated with generic type (ViewFuncT) -> ViewFuncT,
the type variable ViewFuncT = TypeVar(…) must be instantiated to
the *same* type in both places.  This amounts to a claim that the
decorator preserves the signature of the view function, which is not
the case for decorators that add a user_profile parameter.

The corrected annotations enforce no particular relationship between
the input and output signatures, which is not the ideal type we might
get if mypy supported variadic generics, but is better than enforcing
a relationship that is guaranteed to be wrong.

This removes a bunch of ‘# type: ignore[call-arg] # mypy doesn't seem
to apply the decorator’ annotations.  Mypy does apply the decorator,
but the decorator’s incorrect annotation as signature-preserving made
it appear as if it didn’t.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 3916ea23a9 python: Combine some split import groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:54:11 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
sahil839 81c28c1d3e realm: Allow only organization owners to deactivate a realm.
We now allow only organization owners to deactivate a realm.

'require_realm_owner' decorator has been added for this purpose.
2020-06-10 17:33:02 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Mateusz Mandera 200ce821a2 user_activity: Put client id instead of name in event dicts.
This saves the completely unnecessary work of mapping the Client name
to its ID.  Because we had in-process caching of the immutable Client
objects, this isn't a material performance win, but it will eventually
let us delete that caching logic and have a simpler system.
2020-05-29 15:19:55 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Anders Kaseorg 4362cceffb portico: Add setting to put Google Analytics on selected portico pages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-11 23:22:50 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Mateusz Mandera 85df6201f6 rate_limit: Move functions called by external code to RateLimitedObject. 2020-03-22 18:42:35 -07:00
Steve Howell 626ad0078d tests: Add uuid_get and uuid_post.
We want a clean codepath for the vast majority
of cases of using api_get/api_post, which now
uses email and which we'll soon convert to
accepting `user` as a parameter.

These apis that take two different types of
values for the same parameter make sweeps
like this kinda painful, and they're pretty
easy to avoid by extracting helpers to do
the actual common tasks.  So, for example,
here I still keep a common method to
actually encode the credentials (since
the whole encode/decode business is an
annoying detail that you don't want to fix
in two places):

    def encode_credentials(self, identifier: str, api_key: str) -> str:
        """
        identifier: Can be an email or a remote server uuid.
        """
        credentials = "%s:%s" % (identifier, api_key)
        return 'Basic ' + base64.b64encode(credentials.encode('utf-8')).decode('utf-8')

But then the rest of the code has two separate
codepaths.

And for the uuid functions, we no longer have
crufty references to realm.  (In fairness, realm
will also go away when we introduce users.)

For the `is_remote_server` helper, I just inlined
it, since it's now only needed in one place, and the
name didn't make total sense anyway, plus it wasn't
a super robust check.  In context, it's easier
just to use a comment now to say what we're doing:

    # If `role` doesn't look like an email, it might be a uuid.
    if settings.ZILENCER_ENABLED and role is not None and '@' not in role:
        # do stuff
2020-03-11 14:18:29 -07:00
Mateusz Mandera 89394fc1eb middleware: Use request.user for logging when possible.
Instead of trying to set the _requestor_for_logs attribute in all the
relevant places, we try to use request.user when possible (that will be
when it's a UserProfile or RemoteZulipServer as of now). In other
places, we set _requestor_for_logs to avoid manually editing the
request.user attribute, as it should mostly be left for Django to manage
it.
In places where we remove the "request._requestor_for_logs = ..." line,
it is clearly implied by the previous code (or the current surrounding
code) that request.user is of the correct type.
2020-03-09 13:54:58 -07:00
Mateusz Mandera 0255ca9b6a middleware: Log user.id/realm.string_id instead of _email. 2020-03-09 13:54:58 -07:00
Tim Abbott ccf63ac66b decorators: Restructure get_client_name interface.
Previously, get_client_name was responsible for both parsing the
User-Agent data as well as handling the override behavior that we want
to use "website" rather than "Mozilla" as the key for the Client object.

Now, it's just responsible for User-Agent, and the override behavior
is entirely within process_client (the function concerned with Client
objects).

This has the side effect of changing what `Client` object we'll use
for HTTP requests to /json/ endpoints that set the `client` attribute.
I think that's in line with our intent -- we only have a use case for
API clients overriding the User-Agent parsing (that feature is a
workaround for situations where the third party may not control HTTP
headers but does control the HTTP request payload).

This loses test coverage on the `request.GET['client']` code path; I
disable that for now since we don't have a real use for that behavior.

(We may want to change that logic to have Client recognize individual
browsers; doing so requires first using a better User-Agent parsing
library).

Part of #14067.
2020-03-08 14:19:50 -07:00
Tim Abbott 1ea2f188ce tornado: Rewrite Django integration to duplicate less code.
Since essentially the first use of Tornado in Zulip, we've been
maintaining our Tornado+Django system, AsyncDjangoHandler, with
several hundred lines of Django code copied into it.

The goal for that code was simple: We wanted a way to use our Django
middleware (for code sharing reasons) inside a Tornado process (since
we wanted to use Tornado for our async events system).

As part of the Django 2.2.x upgrade, I looked at upgrading this
implementation to be based off modern Django, and it's definitely
possible to do that:
* Continue forking load_middleware to save response middleware.
* Continue manually running the Django response middleware.
* Continue working out a hack involving copying all of _get_response
  to change a couple lines allowing us our Tornado code to not
  actually return the Django HttpResponse so we can long-poll.  The
  previous hack of returning None stopped being viable with the Django 2.2
  MiddlewareMixin.__call__ implementation.

But I decided to take this opportunity to look at trying to avoid
copying material Django code, and there is a way to do it:

* Replace RespondAsynchronously with a response.asynchronous attribute
  on the HttpResponse; this allows Django to run its normal plumbing
  happily in a way that should be stable over time, and then we
  proceed to discard the response inside the Tornado `get()` method to
  implement long-polling.  (Better yet might be raising an
  exception?).  This lets us eliminate maintaining a patched copy of
  _get_response.

* Removing the @asynchronous decorator, which didn't add anything now
  that we only have one API endpoint backend (with two frontend call
  points) that could call into this.  Combined with the last bullet,
  this lets us remove a significant hack from our
  never_cache_responses function.

* Calling the normal Django `get_response` method from zulip_finish
  after creating a duplicate request to process, rather than writing
  totally custom code to do that.  This lets us eliminate maintaining
  a patched copy of Django's load_middleware.

* Adding detailed comments explaining how this is supposed to work,
  what problems we encounter, and how we solve various problems, which
  is critical to being able to modify this code in the future.

A key advantage of these changes is that the exact same code should
work on Django 1.11, Django 2.2, and Django 3.x, because we're no
longer copying large blocks of core Django code and thus should be
much less vulnerable to refactors.

There may be a modest performance downside, in that we now run both
request and response middleware twice when longpolling (once for the
request we discard).  We may be able to avoid the expensive part of
it, Zulip's own request/response middleware, with a bit of additional
custom code to save work for requests where we're planning to discard
the response.  Profiling will be important to understanding what's
worth doing here.
2020-02-13 16:13:11 -08:00
Tim Abbott 7032f49f8e exceptions: Move default json_unauthorized string to response.py.
This small refactor should make it easier to reuse this exception for
other situations as well.
2020-02-05 15:40:10 -08:00
Mateusz Mandera cb71a6571e rate_limiter: Rename 'all' domain to 'api_by_user'. 2020-02-02 19:15:13 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Mateusz Mandera d691c249db api: Return a JsonableError if API key of invalid format is given. 2020-01-03 16:56:42 -08:00
Matheus Melo 31558cb8b9 decorator: Extract require_user_group_edit_permission.
We move the check that the user is a member or admin inot this
decorator.

This name better communicates that this may do other checks beyond
just verifying the policy.
2019-11-18 15:13:29 -08:00
Matheus Melo 21ed834101 decorator: Extract OrganizationAdministratorRequired common exception.
This eliminates significant code duplication of error messages for
situations where an organization administrator is required.
2019-11-18 15:10:56 -08:00
Matheus Melo c96762b7a9 settings: Add setting for who can edit user groups.
Fixes #12380.
2019-11-03 16:45:13 -08:00
Anders Kaseorg cb0652071f zulip_otp_required: Avoid _user_is_authenticated compatibility shim.
This private shim is not needed with Django ≥ 1.10, and was removed
from django-otp 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Eeshan Garg 8e10ab282a webhooks: Log unexpected webhook events separately.
This change serves to declutter webhook-errors.log, which is
filled with too many UnexpectedWebhookEventType exceptions.

Keeping UnexpectedWebhookEventType in zerver/lib/webhooks/common.py
led to a cyclic import when we tried to import the exception in
zerver/decorators.py, so this commit also moves this exception to
another appropriate module. Note that our webhooks still import
this exception via zerver/lib/webhooks/common.py.
2019-07-22 18:20:53 -07:00
Mohit Gupta db3d81613b decorator: Refactor @require_non_guest_human_user decorator.
Rename @require_non_guest_human_user to @require_member_or_admin.
This is a refactor commit prior to introduction of Administrator
Bots.
2019-06-18 17:11:58 -07:00
Mateusz Mandera f73600c82c rate_limiter: Create a general rate_limit_request_by_entity function. 2019-05-30 16:50:11 -07:00
Mateusz Mandera 40763070b7 rate_limiter: Generalise some rate_limit_user code.
We create rate_limit_entity as a general rate-limiting function for
RateLimitedObjects, from code that was possible to abstract away from
rate_limit_user and that will be used for other kinds of rate limiting.
We make rate_limit_user use this new general framework from now.
2019-05-01 12:54:32 -07:00
Eeshan Garg a73e8109b7 webhooks: Remove the legacy GitHub integration.
The github-services model for how GitHub would send requests to this
legacy integration is no longer available since earlier in 2019.
Removing this integration also allows us to finally remove
authenticated_api_view, the legacy authentication model from 2013 that
had been used for this integration (and other features long since
upgraded).

A few functions that were used by the Beanstalk webhook are moved into
that webhook's implementation directly.
2019-04-27 15:13:44 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Mateusz Mandera 5f88406133 rate_limit tests: Cover RateLimiterLockingException case in rate_limit_user. 2019-03-18 11:16:58 -07:00
Rafid Aslam 9038230de8 decorator.py: Add `max_int_size` param to `to_non_negative_int()`.
Add `max_int_size` parameter to `to_non_negative_int()` in
decorator.py so it will be able to validate that the integer doesn't
exceed the integer maximum limit.

Fixes #11451
2019-03-01 11:16:46 -08:00
Anders Kaseorg f0ecb93515 zerver core: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:41:24 -08:00
Tim Abbott 2adb0734d1 decorator: Set an error code for invalid API key errors.
This should make it easily for mobile/terminal apps to handle
situations like the user's API key changing.

Also fix the fact we were incorrectly using a 400, not 401, status
code for this case.
2019-01-05 11:29:34 -08:00
Tim Abbott 4eb80bf3b4 decorators: Don't attempt to rate-limit AnonymousUser.
This fixes our support for sending browser errors to the server for
portico pages in production, which previously hit a rate-limiter
exception.
2019-01-03 15:17:50 -08:00
Tim Abbott 142d9cb63f report: Allow error-reporting views from unauthed users.
This should make it possible for blueslip error reports to be sent on
our logged-out portico pages, which should in turn make it possible to
debug any such issues as they occur.
2018-12-16 15:44:48 -08:00
Tim Abbott 5d5b1afd2d rate_limiter: Improve handling of deadlocks.
We should rate-limit users when our rate limiter deadlocks trying to
increment its count; we also now log at warning level (so it doesn't
send spammy emails) and include details on the user and route was, so
that we can properly investigate whether the rate-limiting on the
route was in error.
2018-12-12 12:19:01 -08:00
Tim Abbott 15d4b71e2e decorator: Skip rate limiting when accessing user uploads.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).

Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).

So we skip running our current rate limiting algorithm for these views.
2018-12-11 12:43:21 -08:00
Tim Abbott 88368996fb decorator: Rename remote_server_request to skip_update_user_activity.
This makes it reasonable to call process_client from other contexts,
like /server_settings, where we don't have an authenticated
UserProfile object.
2018-12-11 11:30:11 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott 9e06ab45bf webhooks: Fix HttpResponse with notify_bot_owner_on_invalid_json.
Apparently, there was a bug in notify_bot_owner_on_invalid_json, where
we didn't reraise the JsonableError.

We fix this with a refactoring that makes the exception layering
clearer as well.
2018-12-05 15:17:31 -08:00
Eeshan Garg b79213d260 webhooks: Notify bot owner on invalid JSON.
There are only a handful of non-JSON webhooks that wouldn't
benefit from the notify_bot_owner_on_invalid_json feature.

Specifically, these are the webhooks where the third-party product
uses another format, whether it be HTML form-encoded, XML, or
something else.

Tweaked by tabbott to correc the list of excluded webhooks.
2018-11-20 15:59:09 -08:00
Eeshan Garg d9958610a4 webhook_decorator: Support notifying bot owner on invalid JSON.
Our webhook-errors.log file is riddled with exceptions that are
logged when a webhook is incorrectly configured to send data in
a non-JSON format. To avoid this, api_key_only_webhook_view
now supports an additional argument, notify_bot_owner_on_invalid_json.
This argument, when True, will send a PM notification to the bot's
owner notifying them of the configuration issue.
2018-11-14 22:35:56 -08:00
Rishi Gupta 7dc021a6b2 billing: Improve error message for require_billing_access. 2018-11-01 14:17:36 -07:00
Vishnu Ks 677f34c2b9 billing: Use require_billing_access decorator in JSON endpoints. 2018-11-01 12:45:29 -07:00
neiljp (Neil Pilgrim) a2767df51d mypy: Enforce strict-optional checking of decorator.py. 2018-10-25 07:24:47 -05:00
Tim Abbott 25b646feaf decorator: Fix error message for accessing deactivated accounts.
The previous message was potentially a lot more ambiguous about
whether this was something about presence.  "Deactivated" makes it
explicit that some action was taken to deactivate the account.
2018-08-09 15:58:31 -07:00
Tim Abbott a124db5f7b decorator: Check for deactivated realm before deactivated user.
Structurally, this is the right order-of-access for data.
2018-08-09 15:57:18 -07:00
Umair Khan bf740f9232 2FA: Add zulip_otp_required decorator.
We need to add this because otp_required doesn't play well with tests.
2018-05-23 15:46:57 -07:00
Umair Khan 0b1e25f453 2FA: Integrate with login feature. 2018-05-23 15:46:57 -07:00
Aditya Bansal 993d50f5ab zerver: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Tim Abbott 508dc5b6ed decorators: Add new decorators for guest users.
These decorators will be part of the process for disabling access to
various features for guest users.

Adding this decorator to the subscribe endpoint breaks the guest users
test we'd just added for the subscribe code path; we address this by
adding a more base-level test on filter_stream_authorization.
2018-05-04 10:25:52 -07:00
Tim Abbott d5946de718 decorator: Add nocoverage comments for rate_limit decorator.
We've already got a bunch of other comments on work we need to do for
this decorator and an open issue that will ensure we at some point
rework this and add tests for it.  In the meantime, I'd like to lock
down the rest of decorator.py at 100% coverage.

Fixes #1000.
2018-04-25 22:37:12 -07:00
Tim Abbott 18e7ef23fc decorator: Add a test to more fully cover require_post.
We also add a nocoverage for what is currently an impossible code
path.
2018-04-25 22:37:09 -07:00
Tim Abbott 2afec13074 decorator: Don't require coverage on @asynchronous @csrf_exempt case.
This line is potentially valuable if we ever do more with this code
path, and doesn't really cost us anything.
2018-04-25 22:36:48 -07:00
Tim Abbott dfb946d84b decorator: Test error cases for authenticated_rest_api_view.
We now have 100% coverage on this important function.
2018-04-25 22:36:48 -07:00
Tim Abbott 2ac67a9c2f decorator: Add nocoverage markings for a few functions. 2018-04-25 22:02:09 -07:00
Tim Abbott e78b11e920 decorator: Move flexible_boolean to github webhook.
We don't really intend to use this hacky function elsewhere in the
codebase, so it's best to move it out of a core file.
2018-04-25 21:59:48 -07:00
Tim Abbott 2fa58fe9ad decorator: Fix exception format for invalid API key.
This exception class was clearly missing the part where `role` gets
stored, which was intended to be inherited from
InvalidZulipServerError.

This fixes an unnecessary 500 error in the push notifications bouncer.
2018-04-25 21:44:31 -07:00
Eeshan Garg 8158342ad3 decorators: Log webhook error payloads in authenticated_rest_api_view.
This completes the effort to ensure that all of our webhooks that do
parsing of the third-party message format log something that we can
use to debug cases where we're not parsing the payloads correctly.
2018-04-23 14:14:42 -07:00
Shubham Dhama b26c38bc47 analytics: Make stats of all realms accessible to server admins.
In this commit:
Two new URLs are added, to make all realms accessible for server
admins. One is for the stats page itself and another for getting
chart data i.e. chart data API requests.
For the above two new URLs corresponding two view functions are
added.
2018-04-18 11:06:50 -07:00
Shubham Dhama 40dc48a033 decorator: Add decorator for checking whether user is server admin.
This is just variabnt of `require_server_admin` for JSON/api views.
2018-04-18 11:01:46 -07:00
Tim Abbott 5ddf2614f0 uploads: Add new way of querying for mobile uploads endpoint.
This extends the /user_uploads API endpoint to support passing the
authentication credentials via the URL, not the HTTP_AUTHORIZATION
headers.  This is an important workaround for the fact that React
Native's Webview system doesn't support setting HTTP_AUTHORIZATION;
the app will be responsible for rewriting URLs for uploaded files
directly to add this parameter.
2018-04-13 17:51:45 -07:00
Eeshan Garg f839d528d8 decorators: Log custom HTTP headers in webhook requests.
This aids in debugging because many of our Git-based integrations
use custom HTTP headers to indicate the type of event/payload.
2018-03-30 13:37:35 -07:00
Tim Abbott 209c813424 decorator: Improve error message for a deactivated organization. 2018-03-16 16:59:02 -07:00
Tim Abbott 34e165c100 webhooks: Fix passing client string to authenticated webhook API views.
This fixes a regression in 93678e89cd
and a4979410f9, where the webhooks using
authenticated_rest_api_view were migrated to a new model that didn't
include setting a custom Client string for the webhook.

When restoring these webhooks' client strings, we also fix places
where the client string was not capitalized the same was as the
product's name.
2018-03-16 15:43:19 -07:00
neiljp (Neil Pilgrim) 88046f815a mypy: Rewrite zulip_login_required annotations in terms of ViewFuncT. 2018-03-15 14:33:56 -07:00
neiljp (Neil Pilgrim) 8edb47e212 mypy: Fully use ViewFuncT in decorators.py; remove WrappedViewFuncT.
Many declarations were previously annotated with
Callable[..., HttpResponse]; this is equivalent to ViewFuncT, so here we
switch to it.

To enable this migration, the WrappedViewFuncT alias is removed; this is
equivalent to the simple & legible Callable[[ViewFuncT], ViewFuncT], so
for relatively no space change, a clearer return type is possible.
2018-03-15 14:33:56 -07:00
neiljp (Neil Pilgrim) 80a7c16baf minor: Remove unused WrapperT from zerver/decorator.py. 2018-03-15 14:33:28 -07:00
neiljp (Neil Pilgrim) 3dbe772b50 mypy: Migrate some Callable[..., HttpResponse] to ViewFuncT in decorator.py. 2018-03-15 14:33:27 -07:00
neiljp (Neil Pilgrim) 17937175ac mypy: Centralize ViewFuncT definition into new file zerver/lib/types.py.
Originally was going to centralize this in zerver/lib/request.pyi, but this
file is not visible at run-time, being only a stub. The matching request.py
file seemed inappropriate, as it doesn't actually use ViewFuncT.
2018-03-15 14:16:40 -07:00
Tim Abbott e2a6541133 i18n: Fix use of 'realm administrator' in translated strings.
These are user-facing and thus should refer to being an "organization
administrator".
2018-03-07 17:15:29 -08:00
Eeshan Garg 75ce1261c1 webhooks/github: Log payloads that aren't handled properly.
Webhook functions wrapped by the decorator:

@authenticated_api_view(is_webhook=True)

now log payloads that cause exceptions to webhook-errors.log.

Note that authenticated_api_view is only used by webhooks/github
and not anywhere else.
2018-03-03 15:30:36 -08:00
Eeshan Garg 5069683cad decorators: Refactor the webhook_logger code.
Just to make the code a bit cleaner and to be able to reuse the
same logging code in other decorator functions.
2018-03-02 11:56:20 -08:00
Dennis Ludl 895a675f4c decorators: Fix email check in access_user_by_api_key case insensitive.
In Zulip, email addresses should always be treated as
case-insensitive; this code path incorrectly assumed the email input
by the user had the correct case.

Discussed in:

https://chat.zulip.org/#narrow/stream/issues/subject/Mobile.20Apps.3A.20Sometimes.20don't.20load.20data/near/461062

Commit message tweaked by tabbott.
2018-01-23 10:04:03 -08:00
Xavier Cooney 5b2819645e mypy: Use Python 3 type syntax in decorator.py. 2017-12-26 08:29:30 -05:00
Greg Price b830b446f1 logging: Reduce `create_logger` to new `log_to_file`.
The name `create_logger` suggests something much bigger than what this
function actually does -- the logger doesn't any more or less exist
after the function is called than before.  Its one real function is to
send logs to a specific file.

So, pull out that logic to an appropriately-named function just for
it.  We already use `logging.getLogger` in a number of places to
simply get a logger by name, and the old `create_logger` callsites can
do the same.
2017-12-12 17:17:08 -08:00
Greg Price b6491fd52a logging: Eliminate log_level from create_logger.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`.  Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
2017-12-12 17:17:08 -08:00
rht a1cc720860 zerver: Use Python 3 syntax for typing.
Tweaked by tabbott to fix some minor whitespace errors.
2017-11-28 16:49:36 -08:00
Steve Howell da2744812f Don't use force_bytes() in decorator.py.
In python3 base64.b64decode() can take an ASCII string, and any
legit data will be ASCII.  If you pass in non-ASCII data, the
function will properly throw a ValueError (verified in python3 shell).

	>>> s = '안녕하세요'
	>>> import base64
	>>> base64.b64decode(s)
	Traceback (most recent call last):
	  File "/srv/zulip-py3-venv/lib/python3.4/base64.py", line 37, in _bytes_from_decode_data
		return s.encode('ascii')
	UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128)

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "<stdin>", line 1, in <module>
	  File "/srv/zulip-py3-venv/lib/python3.4/base64.py", line 83, in b64decode
		s = _bytes_from_decode_data(s)
	  File "/srv/zulip-py3-venv/lib/python3.4/base64.py", line 39, in _bytes_from_decode_data
		raise ValueError('string argument should contain only ASCII characters')
	ValueError: string argument should contain only ASCII characters
2017-11-09 10:43:19 -08:00
neiljp (Neil Pilgrim) 790cd5e7c8 mypy: Specify generic type parameters in cachify decorator.
Use of the decorator in event_queue.py suggests concrete return type,
for application of copy() function.
2017-11-08 12:40:40 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
rht 19bd335cbb Change urllib import to be Python 3-specific. 2017-11-07 10:46:42 -08:00
Steve Howell 5be13fb066 Add cachify decorator. 2017-11-07 10:36:02 -08:00
rht 8990b1046d zerver: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
Greg Price 6b9d294873 auth: Set user_activity `query` nicely for several auth views.
This gets used when we call `process_client`, which we generally do at
some kind of login; and in particular, we do in the shared auth
codepath `login_or_register_remote_user`.  Add a decorator to make it
easy, and use it on the various views that wind up there.

In particular, this ensures that the `query` is some reasonable
constant corresponding to the view, as intended.  When not set, we
fall back in `update_user_activity` on the URL path, but in particular
for `log_into_subdomain` that can now contain a bunch of
request-specific data, which makes it (a) not aggregate properly, and
(b) not even fit in the `CHARACTER VARYING(50)` database field we've
allotted it.
2017-11-04 19:27:00 -07:00
Greg Price b2d35872ed decorator: Set the user_activity query in another case.
Now that this is a little less mysterious-looking to do,
let's do it in this spot too.
2017-11-04 19:27:00 -07:00
Greg Price 1a540241a9 require_server_admin: Cut a redundant bit of request-mutation.
The only place this attribute is used is in `update_user_activity`,
called only in `process_client`, which won't happen if we end up
returning a redirect just below.  If we don't, we go and call
`add_logging_data` just after, which takes care of this already.
2017-11-04 19:27:00 -07:00
Greg Price b871aa9622 user_activity: Allow passing the `query` more directly.
This won't work for all call paths without deeper refactoring,
but for at least some paths we can make this more direct -- function
arguments, rather than mutating a request attribute -- so it's easier
to see how the data is flowing.
2017-11-04 19:27:00 -07:00
neiljp (Neil Pilgrim) 63335fd8c5 mypy: Correct return_success_on_head_request typing.
Also switch to python3 type annotations.
2017-10-29 17:10:13 -07:00
neiljp (Neil Pilgrim) 0781237b1f mypy: Improve typing to profile.py & remove FuncT from decorator.py.
FuncT was unused in decorator.py, and only imported into profile.py.
The @profiled decorator is now more strongly typed on return-type.
Annotations were converted to python3 format.
2017-10-29 17:10:13 -07:00
neiljp (Neil Pilgrim) 1cc0a3cc67 mypy: Improve constraint on return types in statsd_increment decorator. 2017-10-29 17:10:13 -07:00
neiljp (Neil Pilgrim) 232bcd98b3 mypy: Improve return type of internal_notify_view decorator. 2017-10-28 08:57:49 -07:00
neiljp (Neil Pilgrim) 5c3af83229 mypy: Improve return type of api_key_only_webhook_view decorator. 2017-10-28 08:57:49 -07:00
Tim Abbott be619fe881 lint: Wrap many very long lines in the Python codebase.
This decreases the maximum line length in our Python codebase to 130.
2017-10-26 17:31:58 -07:00
Greg Price c9457d4af0 subdomains: Refactor check_subdomain to a clearer interface.
Now that every call site of check_subdomain produces its second
argument in exactly the same way, push that shared bit of logic
into a new wrapper for check_subdomain.

Also give that new function a name that says more specifically what
it's checking -- which I think is easier to articulate for this
interface than for that of check_subdomain.
2017-10-26 10:29:17 -07:00
Greg Price 7bec0a29c4 subdomains: Simplify out check_subdomain from an uncommon use.
This also replaces an implicit use of "" for
Realm.SUBDOMAIN_FOR_ROOT_DOMAIN.
2017-10-26 10:29:17 -07:00
Tim Abbott 1ab2ca5986 subdomains: Extract zerver.lib.subdomains library.
These never really belonged with the rest of zerver.lib.utils.py, and
having a separate library makes it easier to enforce full test
coverage.
2017-10-18 22:27:48 -07:00
Tim Abbott 298c59f7fd push_notifications: Fix error message for unregistered bouncer.
Previously, we were just returning a JSON error to the client, when it
was a server problem.

Fixes #6639.
2017-10-11 19:09:24 -07:00
Tim Abbott 5a27a4cf1c decorator: Improve wrong-subdomain logging output.
Often, it's helpful to see both values in errors.
2017-10-02 13:31:20 -07:00
rht 2949d1c1e8 zerver: Remove the rest of absolute_import. 2017-09-27 10:02:39 -07:00
Aditya Bansal d9c9bfe7f6 logger: Add new create_logger abstraction to simplify logging.
This deduplicates a ton of Python logger-creation code to use a single
standard implementation, so we can avoid copy-paste problems.
2017-08-27 18:31:53 -07:00
Tim Abbott 70e16da81c decorator: Fix request.user handling of remote servers.
The refactor in b46af40bd3 didn't
correctly translate the code for managing request.user and
request._email, resulting in requests for the push notification
bouncer being rejected with this exception:

AttributeError: 'AnonymousUser' object has no attribute 'rate_limits'
2017-08-27 16:35:17 -07:00
Tim Abbott eeabed9119 models: Add new get_user_profile_by_api_key helper.
This results in a slight performance increase.
2017-08-24 23:17:08 -07:00
Tim Abbott ffafa05bd5 decorator: Remove unnecesary import. 2017-08-24 23:16:31 -07:00
Tim Abbott 8b0c0c33b5 login: Fix logging for login/register actions.
Previously, Zulip's server logs would not show which user or client
was involved in login or user registration actions, which made
debugging more annoying than it needed to be.
2017-08-24 16:29:27 -07:00
Tim Abbott 5206dad373 decorator: Fix is_browser_view handling of e.g. mobile apps.
We may end up with some endpoints that could be the browser or the
mobile apps, and we want the right default to happen here.
2017-08-24 16:26:57 -07:00
Tim Abbott 7ffc107061 decorator: Rename is_json_view to is_browser_view.
This improves clarity, now that we're planning to start logging in the
login/registration pages, which aren't browser views.
2017-08-24 16:25:08 -07:00
Tim Abbott fd1b8c7a3c decorator: Fix SOCKET requests being broken with REALMS_HAVE_SUBDOMAINS.
This code path broken in a recent refactoring meant that sending
messages via websockets didn't work if REALMS_HAVE_SUBDOMAINS.
2017-08-15 19:22:17 -07:00
Tim Abbott 93aa28ed08 decorator: Clean up ugly validation comment. 2017-08-15 10:55:32 -07:00
Tim Abbott 5a63ea0170 decorator: Extract validate_account_and_subdomain and deduplicate.
This fixes the significant duplication of code between the
authenticate_log_and_execute_json code path and the `validate_api_key`
code path.

These's till a bit of duplication, in the form of `process_client` and
`request._email` interactions, but it is very minor at this point.
2017-08-15 10:54:23 -07:00
Tim Abbott 127222e7e5 decorator: Clean up process_client call with positional argument. 2017-08-15 10:54:23 -07:00
Tim Abbott 0262609c3d decorator: Use validate_api_key in api_key_only_webhook_view code path.
This completes the effort to deduplicate the main code of our
authentication decorators.
2017-08-15 10:54:23 -07:00
Tim Abbott 319ea62cd9 decorator: Extract zerver/lib/profile.py.
This lets us remove this debugging function from our core
authentication codebase.
2017-08-15 10:54:23 -07:00
Tim Abbott 077c05fc78 decorator: Remove ancient client_name hack for iOS.
The old iOS app has been gone from the app store for 8 months, never
had a huge userbase, and its latest version didn't need this hack.  So
this code is unlikely to do anything in the future; remove it to
declutter our authentication decorators codebase.
2017-08-15 10:54:23 -07:00
Tim Abbott b46af40bd3 decorator: Refactor more code into validate_api_key. 2017-08-15 10:54:23 -07:00
Tim Abbott 5d2d939310 validate_api_key: Stop using 'profile' local variable. 2017-08-15 10:54:23 -07:00
Tim Abbott 25ec30dcf1 decorator: Deduplicate validate_api_key by using new helper function.
This should eliminate a bunch of duplicated code between these two
code paths.
2017-08-15 10:54:23 -07:00
Tim Abbott 5b8d04e18f decorator: Remove now-unnecessary cast. 2017-08-15 10:54:23 -07:00
Tim Abbott 5ebe507c9b decorator: Fix check for incoming webhook bots and move later
The check itself was correct, but the error message was in fact the
opposite of what this check is for.  In other words, the only things
these users can do is post messages, and the error message when you
tried to do something else was to tell you that the user can't post
messages.
2017-08-15 10:54:23 -07:00
Tim Abbott 3c37795675 decorator: Add localhost/Tornado hack to access_user_by_api_key.
This should make it possible to share that code with other code paths.
2017-08-15 10:54:23 -07:00
Tim Abbott 1144eed48b decorator: Clean up 'profile' variable reuse. 2017-08-15 10:54:23 -07:00
Tim Abbott a2011b5a48 decorator: Simplify control flow in validate_api_key.
This technically changes the behavior in the case that
!settings.ZILENCER_ENABLED but is_remote_zulip_server(role).
Fortunately, that case is mostly irrelevant (in that remote zulip
servers is a Zilencer feature).  The old behavior was also probably
slightly wrong, in that you'd get a zilencer-specific error message in
that case.
2017-08-15 10:52:46 -07:00
Tim Abbott a6647f335f decorator: Reorganize remote Zulip server code into one block. 2017-08-15 10:51:29 -07:00
Tim Abbott 052e173aae decorator: Simplify invalid API key error messages. 2017-08-15 10:51:29 -07:00
Tim Abbott 928a93bb9a decorator: Swap order of remote server if statements. 2017-08-15 10:51:29 -07:00
Tim Abbott 3157618b80 decorators: Extract access_user_by_api_key. 2017-08-15 10:51:29 -07:00
Umair Khan ef1b880082 rate_limiter: Upgrade api_calls_left to generic API. 2017-08-02 18:01:39 -07:00
Umair Khan 81e2128774 rate_limiter: Upgrade incr_ratelimit to generic API. 2017-08-02 18:01:39 -07:00
Umair Khan dd281367ae rate_limited: Upgrade is_ratelimited to generic API. 2017-08-02 18:01:39 -07:00
Tim Abbott 04729a0e79 mypy: Remove some now-unused type: ignores. 2017-07-27 16:12:26 -07:00
Greg Price a5597e91a1 exceptions: Move zerver/exceptions.py under zerver/lib/.
Seems like a more appropriate place for it.  Preparation for
moving a bit more into that file.
2017-07-24 16:41:22 -07:00
Greg Price e18baff32c JsonableError: Rename message from `error` to `msg`.
The whole thing is an error, so "message" is a more apt word for the
error message specifically.  We abbreviate that as `msg` in the actual
HTTP responses and in the signatures of `json_error` and friends, so
do the same here.
2017-07-24 16:41:22 -07:00
Eeshan Garg a24ea56c53 decorator: Handle invalid JSON in api_key_only_webhook_view.
Exception logging within api_key_only_webhook_view fails when
ValueError is raised if the request.body passed to ujson.loads
isn't valid JSON. In this case, we now just convert the payload
to a string and log that. This allows us to inspect JSON payloads
that aren't being decoded properly.
2017-07-19 10:42:35 -07:00
Vaida Plankyte 8c4ee83980 decorator.py: Use the singular 'they' pronoun. 2017-07-05 09:27:43 -07:00
Umair Khan 1e9c87855c Django 1.11: is_authenticated is now a property. 2017-06-13 15:04:04 -07:00
Aditya Bansal 4679da87c4 pep8: Add compliance with rule E261 decorator.py. 2017-06-04 09:18:22 -07:00
Ethan d1bd19a1b8 mypy: correct user_passes_test first argument. 2017-05-25 15:41:48 -07:00
Ethan c284d913cc mypy: request.body is bytes, should be str. 2017-05-25 15:41:46 -07:00
Umair Khan 4e2a6834d8 authenticated_rest_api_view: Use is_remote_server.
Using is_remote_server is more readable and future-proof since
it provides a level of abstraction.
2017-05-16 12:26:55 -07:00
Tim Abbott 0b46be2fed Avoid UserActivity logging for RemoteZulipServer requests.
We had a somewhat messy bug where we were sending invalid entries to
the UserActivity queue when using the push notification bouncer.
2017-05-15 17:15:01 -07:00
Eeshan Garg 10dcc99983 zerver/decorators: Log all exceptions raised in api_key_only_webhook_view.
Fixes #4742.
2017-05-13 14:53:47 -07:00
Umair Khan faf190ff34 validate_api_key: Use is_remote_server. 2017-05-11 12:02:26 -07:00
Rishi Gupta e46cbaffa2 email: Remove Mandrill pathways and dependency.
Everything it was doing (send_future_email) can now be done using
ScheduledJob.
2017-05-05 14:38:23 -07:00
Eeshan Garg e87e246fcb zerver/decorator: Set request.client in api_key_only_webhook_view.
Previously, api_key_only_webhook_view passed 3 positional arguments
(request, user_profile, and client) into a function. However, most
of our other auth decorators only pass 2 positional arguments. For
the sake of consistency, we now make api_key_only_webhook_view set
request.client and pass only request and user_profile as positional
arguments.
2017-05-01 23:44:07 -07:00
Maxim Averin 73a1dd63d5 analytics: Refactor legacy 'zulip_internal' decorator.
Rename 'zulip_internal' decorator to 'require_server_admin', add
documentation for 'server_admin', explaining how to give permission
for ./activity page.

Fixes: #1463.
2017-04-22 11:42:02 -07:00
Tim Abbott cddee49e75 Add support infrastructure for push notification bouncer service.
This is an incomplete cleaned-up continuation of Lisa Neigut's push
notification bouncer work.  It supports registration and
deregistration of individual push tokens with a central push
notification bouncer server.

It still is missing a few things before we can complete this effort:
* A registration form for server admins to configure their server for
  this service, with tests.
* Code (and tests) for actually bouncing the notifications.
2017-04-18 23:03:06 -07:00
Tim Abbott ae788b2e98 zilencer: Remove decorators Deployment code. 2017-04-18 23:00:10 -07:00
Tim Abbott cbc731963f decorator: Add support for Django internal_notify_view. 2017-04-18 09:59:07 -07:00
Rishi Gupta e14c940ecc decorator: Add human_users_only decorator.
Applies it to presence.update_active_status_backend as an example of usage.
2017-04-16 12:51:23 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
Tim Abbott 033fd98e5f decorator: Add rate limiting to zulip_login_required. 2017-03-26 13:10:43 -07:00
Tim Abbott 385551ff62 decorator: Add rate-limiting to JSON views. 2017-03-26 13:10:43 -07:00
Tim Abbott 6c4c8178f0 zulip_login_required: Set request._query.
This fixes an exception we had in the user_activity queue processor
when changing email addresses, since the URL containing the
confirmation key was longer than 50 characters.
2017-03-26 13:10:43 -07:00
K.Kanakhin 234a1f8e61 retention-period: Add retention period to front-end admin organization settings.
- Add message retention period field to organization settings form.
- Add css for retention period field.
- Add convertor to not negative int or to None.
- Add retention period setting processing to back-end.
- Fix tests.

Modified by tabbott to hide the setting, since it doesn't work yet.
The goal of merging this setting code now is to avoid unnecessary
merge conflicts in the future.

Part of #106.
2017-03-25 17:57:18 -07:00
Umair Khan 149b0c30df capitalization: Fix Only Basic authentication is supported. 2017-03-08 23:40:57 -08:00
Tim Abbott 75e81253f2 mypy: Work around several new mypy bugs in 0.501. 2017-03-04 15:33:39 -08:00
Rishi Gupta 15d60fa7ed Change now() to timezone.now() throughout codebase.
Change `from django.utils.timezone import now` to
`from django.utils import timezone`.

This is both because now() is ambiguous (could be datetime.datetime.now),
and more importantly to make it easier to write a lint rule against
datetime.datetime.now().
2017-02-28 16:03:37 -08:00
Tim Abbott 20f9c04ab5 decorator: Add logging data to zulip_login_required.
This fixes an issue that many logged=in pages such as /stats did not
correctly report either the connecting client or the user in server
logs.
2017-02-20 21:15:44 -08:00
Tim Abbott c9126e772e decorator: Don't block ZulipElectron name in json views.
This fixes the logging of the ZulipElectron client in server logs.
Message sending is still logged as "website"; that will be fixed soon.
2017-02-17 15:19:10 -08:00
Tim Abbott 2bba7755e3 decorator: Improve user agent parsing. 2017-02-10 23:53:44 -08:00
Tim Abbott bc63407e6a decorator: Cleanup api_key_legacy code readability. 2017-02-10 23:53:44 -08:00
Tim Abbott bb5d81281c mypy: Upgrade to new package name and version 0.571.
Fixes #3448.
2017-02-10 23:53:44 -08:00
Tim Abbott 70af09539f Fix unnecessary traceback in authenticated_rest_api_view.
Apparently, we weren't returning the `json_error`, resulting in users
encountering this condition receiving a 500, rather than the proper
40x error.

This fixes a regresion introduced in 9ae68ade8b.
2017-01-29 12:48:10 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott 22d1aa396b lint: Clean up W503 PEP-8 warning. 2017-01-23 20:50:04 -08:00
Tommy Ip 008663abd8 zerver: Fix bare except clause. 2017-01-09 00:38:31 +00:00
Tim Abbott 0847515203 Fix text_type->Type mypy merge conflict error. 2016-12-29 15:03:37 -08:00
Rishi Gupta 7b057392c6 decorator.py: Add to_utc_datetime converter function. 2016-12-29 14:20:13 -08:00
Juan Verhook cfa9c2eaf2 mypy: Update zerver directory to use Text 2016-12-29 09:12:15 -08:00
Tim Abbott 7e8f8551de decorator: Change /activity to be gated on is_staff.
Zulip doesn't previously make use of the standard Django is_staff flag
(in that the Django admin site is disabled), but since conceptually
the /activity page would be part of the Django admin site if we were
using it (i.e. for server-level administrators), it makes sense to key
off of that rather than the previous, fragile, check for the realm
domain name.
2016-12-13 21:06:27 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
bulat22101 adebc75740 pep8: Fix E502 violations 2016-12-03 10:56:36 -08:00
AZtheAsian c23f9e0df7 pep8: Fix E111 violations 2016-12-01 14:18:11 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Tomasz Kolek d8dee522b6 Fix trello integration by adding handling HEAD confirmation request.
Previously, we rejected the HEAD requests that the trello integration
uses to check if the server accepts the integration.

Add decorator for returning 200 status code if request is HEAD.

Fixes: #2311.
2016-11-18 16:27:16 -08:00
Umair Khan fe1ba6f3eb Django 1.10: Deprecate request.REQUEST. 2016-11-04 10:20:24 -07:00
Steve Howell 403490a95b annotations: Fix to_non_negative_int(). 2016-10-28 13:48:44 -07:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Tim Abbott b5b00e3a36 subdomains: Enforce subdomain checks on every API endpoint.
This ensures that everything is using the correct subdomain for
requests.  While it probably wouldn't be a real security problem for
the wrong subdomain to work, this enforcement is essential to catching
bugs in the product and users' API scripts.
2016-09-27 23:25:07 -07:00
hackerkid ea39fb2556 Add option for hosting each realm on its own subdomain.
This adds support for running a Zulip production server with each
realm on its own unique subdomain, e.g. https://realm_name.example.com.

This patch includes a ton of important features:
* Configuring the Zulip sesion middleware to issue cookier correctly
  for the subdomains case.
* Throwing an error if the user tries to visit an invalid subdomain.
* Runs a portion of the Casper tests with REALMS_HAVE_SUBDOMAINS
  enabled to test the subdomain signup process.
* Updating our integrations documentation to refer to the current subdomain.
* Enforces that users can only login to the subdomain of their realm
  (but does not restrict the API; that will be tightened in a future commit).

Note that toggling settings.REALMS_HAVE_SUBDOMAINS on a live server is
not supported without manual intervention (the main problem will be
adding "subdomain" values for all the existing realms).

[substantially modified by tabbott as part of merging]
2016-09-27 23:24:14 -07:00
Tim Abbott 15f6cc7c84 validate_api_key: Accept the request as an argument.
This is a prerequisite for checking the subdomain of the request.
2016-09-27 21:18:29 -07:00
Tim Abbott 4840e493b2 decorator: Add missing annotation. 2016-08-04 15:57:03 -07:00
Tim Abbott 700e803840 settings: Add new DEBUG_RATE_LIMITING setting.
This exists primarily in order to allow us to mock settings.DEBUG for
the purposes of rate limiting, without actually mocking
settings.DEBUG, which I suspect Django never intended one to do, and
thus caused some very strange test failures (see
https://github.com/zulip/zulip/pull/776 for details).
2016-07-29 19:32:02 -07:00
Eklavya Sharma 09e4e5aea7 Add links after `type: ignore`s. 2016-07-22 13:59:31 -07:00
Eklavya Sharma 674f6999e1 Improve annotations of decorators. 2016-07-22 11:14:33 -07:00