Commit Graph

28467 Commits

Author SHA1 Message Date
Steve Howell 31597cf33e Remove timeout parameter in do_rest_call().
Nobody was setting it.
2018-10-11 16:12:07 -07:00
Steve Howell 69ee84bb14 refactor: Extract build_bot_request().
This fixes a couple things:

    * process_event() is a pretty vague name
    * returning tuples should generally be avoided
    * we were producing the same REST parameters in both
      subclasses
    * relative_url_path was always blank
    * request_kwargs was always empty

Now process_event() is called build_bot_request(),
and it only returns request data,
not a tuple of `rest_operation` and `request_data`.

By no longer returning `rest_operation`, there are
fewer moving parts.  We just have `do_rest_call` make
a POST call.
2018-10-11 16:12:07 -07:00
Steve Howell 16eff75e49 refactor: Simplify how we use base_url.
Before this change, we instantiated base_url into a superclass
of subclasses that returned base_url into a dictionary that
gets returned to our caller.

Now we just pull base_url out of service when we need to make
the REST call.
2018-10-11 16:12:07 -07:00
Steve Howell b89a94f730 Improve errors when we can't connect to a bot server.
We don't overwhelm people with error info when bots
fail to connect or time out.
2018-10-11 16:12:07 -07:00
Steve Howell 3790c469e9 outgoing bots: Report JSON errors to users.
We should arguably report these to bot owners
as well, but this is at least an improvement
over having the server crash.
2018-10-11 16:12:07 -07:00
Steve Howell df4b665658 refactor: Parse JSON from bots in one place.
We move the JSON parsing step into the
higher level function: process_success_response().

In the unlikely event that we'll start integrating
with a solution that doesn't use JSON, we can deal
with that, and for now doing the parsing in one
place will help us make error reporting more
consistent.

In a subsequent commit we'll introduce better
error handling for malformed JSON.
2018-10-11 16:12:07 -07:00
Steve Howell 395d74c08a api docs: Clean up outgoing webhooks section.
I rewrote the section explaining what the
endpoint sends back to the server.  This fixes
a few typos, emphasizes the normal case, and
starts to favor "content" as the key for
content.
2018-10-11 16:12:07 -07:00
Steve Howell 229dd5d861 outgoing webhooks: Get rid of "Success!" prefix.
The earlier code here, if it got a payload with
"response_string" as a key, would prefix the
corresponding value with "Success!".  We just
want the bot to set its own content.

The code is reorganized here so that process_success()
always produces a value keyed by "content" from
incoming data, and then process_success_response()
doesn't do any fancy munging of the data.
2018-10-11 16:12:07 -07:00
Tim Abbott 0a751567a3 upload: Fix missing mypy return type annotation. 2018-10-11 16:11:20 -07:00
Joshua Pan 971cb18cb3 user_settings: Compare new stripped email with old email.
We weren't comparing the newly stripped email with the current
old email, thus adding spaces around an email would result in
an error.
2018-10-11 15:55:32 -07:00
Tim Abbott 8cf104b643 avatar: Allow API authentication for /avatar/ routes.
This makes it feasibly for the mobile apps to correctly render user
avatars generated by the `!avatar()` syntax.
2018-10-11 15:52:29 -07:00
Aditya Bansal 3164f1a9a4 avatar: Rename user_profile to avatar_user_profile.
This is a preparatory commit for upcoming changes to move
/avatar/ to be a logged in or API accessible endpoint.
Basically we rename this variable because the new name is more
appropriate in the situation. Also user_profile will be used to
hold the user_profile of person accessing the endpoint in coming up
commit.
2018-10-11 15:50:37 -07:00
Aditya Bansal 6893f52ad9 thumbnails: Instruct thumbor to sharpen thumbnailed images.
Fixes: #10218.
2018-10-11 15:44:47 -07:00
Aditya Bansal 6e433186a1 thumbnails: Change thumbnail size to be 300px.
Fixes: #10219.
2018-10-11 15:44:47 -07:00
Vishnu Ks 962d72b58b retention: move_messages_to_archive should accept multiple message ids.
This will speed up the scrub realm management command. Calling the
function with a single message_id in a loop was extremely inefficient.
2018-10-11 15:31:12 -07:00
Vishnu Ks 6972de21be management: Add command to scrub a realm of personal data. 2018-10-11 15:30:26 -07:00
Vishnu Ks 2f5a5c2c49 test_classes: Create lear_user helper function. 2018-10-11 15:30:26 -07:00
Vishnu Ks 5bdadc8061 upload: Create function to delete avatar image. 2018-10-11 15:30:26 -07:00
Vishnu Ks 1d94fc7dbb upload: Extract function to delete file. 2018-10-11 15:30:26 -07:00
Steve Howell 4b82326376 hipchat import: Support guest users.
We simplify the code for is_realm_admin
and set is_guest as well.

I verified that build_user() is not used
by Slack/Gitter, so the extra argument there
should be fine.

Fixes #10639
2018-10-11 15:28:58 -07:00
Steve Howell f1e0152d71 minor: Fix indentation from recent ldap commit.
The linter was complaining.
2018-10-11 15:28:58 -07:00
Vishnu Ks 646ba439f7 settings: Remove unused EMAIL_LOG_DIR setting from dev_settings.
This settings is not used anywhere.
2018-10-11 15:12:08 -07:00
Vishnu Ks 6aa4b64dc0 emails: Don't log emails while running test suite.
Modified the tests to ensure 100% coverage.
2018-10-11 15:12:08 -07:00
Vishnu Ks d8c19cb003 models: Move billing models from zilencer to corporate. 2018-10-11 14:54:29 -07:00
Greg Price d77a4c776d ldap: Skip following dubious referrals.
Some admins setting up Zulip's LDAP auth against Active Directory see
a rather baffling error message: "In order to perform this operation a
successful bind must be completed on the connection".  This happens
despite AUTH_LDAP_BIND_DN and auth_ldap_bind_password being set
perfectly correctly, and on a query that the `ldapsearch` command-line
tool performs quite happily.

Empirically, adding a setting like this to /etc/zulip/settings.py
resolves the issue:
  AUTH_LDAP_CONNECTION_OPTIONS = {
      ldap.OPT_REFERRALS: 0
  }

Some useful, concise background on the LDAP "referral" concept is here:
  https://docs.oracle.com/javase/jndi/tutorial/ldap/referral/overview.html
and a pertinent bit of docs for the underlying Python `ldap` client:
  https://www.python-ldap.org/en/latest/faq.html
and some very helpful documentation for Active Directory:
  https://docs.microsoft.com/en-us/windows/desktop/ad/referrals

Based on the docs above, the story appears to be something like this:

 * This server has the information for part of the scope of our query
   -- in particular it happens to have the information we actually want.

 * But there are other areas ("subordinate domains") that our query is
   in principle asking about, and this server doesn't know if there are
   matches there, so it gives us a referral.

 * And by default, python-ldap lets `libldap` run ahead and attempt to
   bind to those referrals and do those queries too -- which raises an
   error because, unlike Microsoft's "LDAP API", it doesn't reuse the
   credentials.

So if we simply skip trying to follow the referrals, there's no
error... and we already have, from the original response, the answer
we actually need.  That's what the `ldap.OPT_REFERRALS` option does.

There may be more complex situations where the referral really is
relevant, because the desired user info is split across servers.  Even
then, unless an anonymous query will be acceptable, there's no point
in letting `libldap` follow the referral and setting this option is
still the right thing.  When someone eventually comes to this bridge,
some code will be required to cross it, by following the referrals.
That code might look a bit like this (unfinished) example:
  https://bugs.launchpad.net/nav/+bug/1209178

Manually tested by tabbott.

Fixes #343, which was effectively a report of the need for this
OPT_REFERRALS setting.

Fixes #349, since with this change, we no longer require tricky manual
configuration to get Active Directory up and running.
2018-10-11 14:52:13 -07:00
Tim Abbott 9054ce278f locale: Fix incorrect use of locale-gen to generate locales.
Previously, we were having issues installing on Debian Stretch with
non-English locales, because `locale-gen` actually doesn't take a
locale as an argument (and thus `locale-gen en_US.UTF-8` did nothing).
We should instead be calling localedef directly.

Thanks to Tom Daff for debugging this.

Fixes #10629.
2018-10-11 14:42:24 -07:00
Jack Zhang d60a088a49 developer docs: Recommend the mypy daemon for running mypy locally.
Resolves #10622.
2018-10-10 16:29:16 -07:00
Tim Abbott c57c4cf703 notifications: Fix push notifications with multiple realms.
Previously, Zulip did not correctly handle the case of a mobile device
being registered with a push device token being registered for
multiple accounts on the same server (which is a common case on
zulipchat.com).  This was because our database `unique` and
`unique_together` indexes incorrectly enforced the token being unique
on a given server, rather than unique for a given user_id.

We fix this gap, and at the same time remove unnecessary (and
incorrectly racey) logic deleting and recreating the tokens in the
appropriate tables.

There's still an open mobile app bug causing repeated re-registrations
in a loop, but this should fix the fact that the relevant mobile bug
causes the server to 500.

Follow-up work that may be of value includes:
* Removing `ios_app_id`, which may not have much purpose.
* Renaming `last_updated` to `data_created`, since that's what it is now.

But none of those are critical to solving the actual bug here.

Fixes #8841.
2018-10-10 16:15:52 -07:00
Tim Abbott 83bcea3917 api: Fix docs to point to /latest/ for integration docs.
Pointing to /stable/ doesn't work, since that article doesn't exist in
the 1.8.0 release series.
2018-10-10 12:19:14 -07:00
Akash Nimare f9179e9903 login: Vertically align the buttons.
This sets the padding top to 13px so that it matches with
the bottom padding.
2018-10-10 11:57:44 -07:00
Akash Nimare f00cd549cd login-page: Make the buttons and labels cleaner.
Changes -
a) Updated the border-radius to 4px for all the buttons.
b) Increased the margins between the labels and inputs.

These changes affect the login and register page's styling.
2018-10-10 11:57:44 -07:00
Akash Nimare 3e72db8c62 devlogin: Update the border-radius of all the buttons.
This commit updates the border-radius of the buttons presents
on the /devlogin page.
2018-10-10 11:57:44 -07:00
Rishi Gupta bf22eefede api docs: Move integration-docs-guide to docs/. 2018-10-09 20:28:44 -07:00
Rishi Gupta 8001d7e84c docs: Move subsystems/documentation to be closer to other docs articles. 2018-10-09 20:28:44 -07:00
Rishi Gupta 29f67014af docs: Clean up user-docs.md. 2018-10-09 20:28:44 -07:00
Akash Nimare da7adb038f thumbnail: Show play button on hovering youtube thumbnail.
This PR adds a play icon on hovering the youtube thumbnails.
Also, shows the pointer cursor on vimeo videos instead of
zoom-in.
2018-10-09 16:12:36 -07:00
Steve Howell c0df049a18 Allow "content" from outgoing webhooks.
We now allow outgoing webhooks to provide us a
"content" field, which is probably a more guessable
name than "response_string", particularly for folks
that use our other bot-related APIs.  And we don't
modify content as we do response_string, i.e. no
"Success!" prefix.

If we're not too concerned about backward compatibility,
we can do a subsequent commit that makes "content"
and "response_string" true synonyms and get rid of
the "Success!" prefix, which was probably accidental
to begin with.
2018-10-09 15:56:24 -07:00
Steve Howell 6c4343c86d refactor: Clean up send_response_message().
This commit starts by changing the third
argument of send_response_message to be a Dict
instead of a string, so that the data can be more
structured going forward.

That change makes the 2nd/3rd parameters both be
dicts, so to be defensive, I now have all the callers
pass in explicit keyword names.  And then I rename
message to message_info, so that the callers have
more clear code.

And that changes the implementation inside of
send_response_message() a bit.

Sorry this commit is a bit coarse, but the intermediate
commits would have been kind of ugly, too.

At the end of the day, it's pretty simple:

    bot_id: never changed
    message_info: just renamed from message
    response_data: is a Dict with the key of "content"

And the innards of send_response_message() are basically
simply dictionary lookups and function calls.
2018-10-09 15:56:24 -07:00
Steve Howell 4956107c53 refactor: Simplify return type for process_success().
There's no reason to return a failure message in
process_success(), since it's implied to be part of
the success codepath.  I didn't look at the full history
of how the strange API evolved, but the second element
of the tuple was clearly noise by the time I got here.
Neither of the subclasses ever set it, and none of the
consumers used it.
2018-10-09 15:56:24 -07:00
Steve Howell f2dd218331 refactor: Inline succeed_with_message().
This two-line function wasn't really carrying its
weight, and it just made it harder to refactor the
overall codepath.

Eliminating the function forces us to mock at a slightly
deeper level, which is probably a good thing for what
the test intends to do.  The deeper mock still verifies that
we're sending the message (good) without digging into
all the details of how we send it (good).

Note that we will still keep around the similarly named
`fail_with_message` helper, which is a lot more useful.
(The succeed/fail scenarios aren't really symmetric here.
For success, there are fewer codepaths that do more complex
things, whereas we have lots and lots of failure codepaths
that all do the same simple thing of replying with a canned
message.)
2018-10-09 15:56:24 -07:00
Steve Howell fa505a1af1 refactor: Have process_success return structured data.
Before this change subclasses of OutgoingWebhookServiceInterface
would return a raw string as the first element of its return
tuple in process_success().  This is not a very flexible
design, as it prevents the bot from passing extra data like
`widget_content`.

It's also possible in the future that we'll want to let outgoing
bots reply directly to senders who mention them on streams, and
again the original design was overly constrained for that.

This commit does not actually change any functionality yet.
2018-10-09 15:56:24 -07:00
Steve Howell 3bb8cbe0c7 minor: Dedup check_send_message() call. 2018-10-09 15:56:24 -07:00
Steve Howell e641036911 minor: Rename var to message_type. 2018-10-09 15:56:24 -07:00
Steve Howell b61612d50b minor: De-duplicate code for client. 2018-10-09 15:56:24 -07:00
Callum Fraser 383f1633e6 api: Add limit of total messages available per request of 5000.
Tweaked by tabbott to use a declared constant rather than just use
5000 in multiple places; this also means we can change the count
without updating translations.

Fixes #10446.
2018-10-09 15:43:21 -07:00
Rishi Gupta 7956c57448 user docs: Add import-from-hipchat. 2018-10-09 15:23:07 -07:00
Rishi Gupta 99eb13a3f7 user docs: Update import docs.
The major changes are:
* Remove the --destroy-rebuild-database option
* Merge the new and existing self-hosted server sections
* Change the wording of the Gitter document to match the Slack one
2018-10-09 15:23:07 -07:00
Rishi Gupta 4265b8e62f user docs: Soften the h1 underline in the sidebar. 2018-10-09 15:23:07 -07:00
Lyla Fischer 3c51328763 user docs: Remove icon-vector for font-awesome. 2018-10-09 14:16:16 -07:00
Pragati Agrawal 749e034863 org settings: Rename "Filter settings" to "Linkifiers".
Fixes: #10551
2018-10-09 08:50:45 -07:00