Tip -- 2.1d - Using the smfUpdateDownloadTable macro ("referback")

---In smf_addin@yahoogroups.com, <rharmelink@...> wrote :

The add-in is intended for ad hoc data retrieval. The way it extracts data is to save a copy of the web page, then to extract any data elements from that saved web page (as opposed to retrieving the web page once for each data item that comes from the web page). However, it only has 1000 slots for storing web pages. After those 1000 slots are full, all requests for data from additional web pages will error out.

I really didn't want people to start building databases of information, simply because they could. That would be unfair to the free data services, and with even people doing it might encourage them to code their web pages so that something like the add-in couldn't access them.

However, having said that, I think the smfUpdateDownloadTable process might be the best way for you to go. That is a macro that fills in a 2-dimensional table -- tickers by data items -- with values instead of formulas that recalculate every time the workbook is opened. Updating is controlled by you -- whenever you run the macro.

If you use the beta version of the add-in, you can even tell it which ranges to update -- entire rows (i.e. tickers), entire columns (i.e. specific data items, or even some range within the table. I sometimes have a column with a "Timestamp" so I know when I last updated the row.

Some information on that macro can be found on the "Tips and FAQS" web page on the add-in web site. Templates are in the FILES area of the Yahoo group, including the template I always start from, because it has everything already set up for the macro:

smfUpdateDownloadTable-Sample.xls

One thing not documented on the "Tips and FAQs" web page is that there is also a "referback" method. That is, using "~~~n~~~" (where "n" is 1, 2, 3, or whatever) says to use the value from that many columns to the left. So, "~~~1~~~" says to replace the data item from the previous column where "~~~1~~~" is. One technique I've used there is to have something like:

TEXT(COLUMNS($E2:Q2)-1,"~~~0~~~")

...where I want to use the value of whatever is in column E in my formula in column Q. Otherwise I needed to hardcode a "~~~12~~~" in the formula. However, if I inserted a new column, I need to make changes to all hardcoded formulas that use column E. By using the "COLUMNS()" function, the referral stays intact if columns are inserted.

So, for example, a formula for column Q of the table might look like:

="(""~~~1~~~""+""~~~2~~~"")/"""&TEXT(COLUMNS($E2:Q2)-1,"~~~0~~~")&"""-1"

I've even evolved the process to the point of assigning the referbacks to defined names, so I can do something like:

="IF(ISNUMBER("&cBid2&"),(MIN("&cStrike2&","&cPrice&")+"&cBid2&")/"&cPrice&"-1,""--"")"

...where cBid2, cStrike2, and cPrice all look like that TEXT() formula above. The defined names make it a lot easier to understand what the formula is referring to. :)

On Wed, Apr 22, 2015 at 8:58 AM, austinmlazar@... wrote:

I have hit a road block in the midst of a very exciting project I was working on, using the RCHGetElementNumber () function I have implemented over 43 important metrics that automatically generate for every ticker entered, I have made 506 rows with the plan to reweight the S&P dynamically with an Expected Alpha based weighting system. After about 145 tickers that worked seamlessly and uploaded all 43 data points within 45 seconds almost every time. Below a certain point it starts to give me error outputs for every data point and I know this isn't right because it worked seamlessly with any random ticker I would put it. I also tried to recreate another sheet thinking maybe the worksheet was overloaded but do I have to create an entirely new workbook for every increment of 145 or are there other fixes to this?

Tip -- 2.1b - Using the smfUpdateDownloadTable macro

---In smf_addin@yahoogroups.com, <rharmelink@...> wrote :

Let me restate [Tip 2.1a] and give some visuals of creating a range to fill using the smfUpdateDownloadTable macro:

1. The upper left hand corner cell of the table needs to have a range name of "Ticker" (see green-shaded cell on diagram). The text in the cell can be anything you want it to be.
2. The cells below the "Ticker" cell should be filled in with Yahoo ticker symbols, one per cell (see yellow-shaded cells on diagram).
3. The cells to the right of the "Ticker" cell should be filled with column titles (see orange-shaded cells on diagram. The text in these cells can be anything you want them to be.
4. The cells above the column titles need to be filled in with SMF add-in formulas or element numbers (see blue-shaded cells on diagram). Use five tildas as a substitute for a ticker symbol. For example, any of the following text strings could be used to get "Market Capitalization" from Yahoo:
   
   941
   RCHGetElementNumber("~~~~~", 941)
   RCHGetTableCell("http://finance.yahoo.com/q/ks?s=~~~~~",1,">Market Cap")
   
   If you place an "X" in the cell of the element definition, it would tell the smfUpdateDownloadTable macro to skip it and leave that column alone. I do this when I want to create a column with my own formulas in, or to use as a divider column in the table.

The macro continues processing rows with ticker symbols (i.e. green-shaded cell) until it runs into an empty cell with no ticker symbol in it. It continues process across the row of element definitions (i.e. blue-shaded cells) until it runs into an empty cell with no element definition in it.

So, this is what it would look like after doing the four steps above:

	941	X	942
Heading0	Heading1	Heading2	Heading3
MMM
IBM
AA

After running the smfUpdateDownloadTable macro, it would look like this:

	941	X	942
Heading0	Heading1	Heading2	Heading3
MMM	58800000		61300000
IBM	1.73E+08		1.88E+08
AA	15230000		23800000

Tip -- 2.1a - Using the smfUpdateDownloadTable macro

---In smf_addin@yahoogroups.com, <rharmelink@...> wrote :

I use the smfUpdateDownloadTable macro for such situations.  This would actually place values into the table, so no recalculation occurs.  When I want "fresh" values, I just rerun the macro.  Here is the documentation for the table setup from the macro's module:

1. The upper left hand corner cell of the table needs to have a range name of "Ticker"
2. The cells below the "Ticker" cell should be filled in with ticker symbols, one per cell
3. The cells to the right of the "Ticker" cell should be filled with column titles
4. The cells above the column titles need to be filled in with SMF add-in formulas or element numbers.  Use five tildas as a substitute for a ticker symbol.  For example, any of the following text strings could be used to get "Market Capitalization" from Yahoo:
   
   941
   RCHGetElementNumber("~~~~~", 941)
   RCHGetTableCell("http://finance.yahoo.com/q/ks?s=~~~~~",1,">Market Cap")

A recent addition to the process is that if the cell entry above a column title is an "x", that column is skipped from any update -- I use this for any calculation columns I want in the middle of my table.  In all of my workbooks, I assign the smfUpdateDownloadTable macro to a keyboard shortcut of Ctrl-Shift-J.

On 10/19/07, rjlabs <rick@...> wrote:
>
> I have a large table of =RCHGetElementNumber() that has historical
> quarterly info that I only need to update sporadically. It takes about
> 15 minutes to update (about 60 tickers x 150 fields).
>
> Is there a way I can control when those functions run and update? I
> want to be able to open the spreadsheet with perhaps stale data and
> not have it update. However I don't want to set calculation to manual
> because other parts of the workbook I want regularly calculating. (Any
> programmer guide or FAQ or embedded docs on this just point me to them.) 

Tip -- Using smfGetCSVFile() to grab Yahoo Historical quotes

It appears that Yahoo once again has an easy-to-grab CSV file of their historical quotes available directly via a URL. See the link below for an example, using the smfGetCSVFile() function. It does use the new EXCEL "spill" feature -- if you don't have that available, all you will see is "Date" for the function, as you would need to array-enter the function. The new "spill" feature automatically expands any array function to fill the range is has data for.

SMF-Sample-smfGetCSVFile-Yahoo-Historical-Quotes.xlsm

Tip: Data retrieval with Python

If you're not familiar with Python, you should learn. It's a powerful programming language that can extract data from websites. And there are many code libraries for doing that retrieval. 

The Google Colaboratory website allows you to easily run Python code from the browser:

https://colab.research.google.com/notebooks/intro.ipynb#recent=true

A good introduction to retrieving Yahoo data is here:

https://medium.com/@realmistic/use-your-computer-to-make-informed-decisions-in-stock-trading-practical-introduction-part-2-832e6f67f590

For example, a program as simple as:

!pip install yfinance

import yfinance as yf

pfe = yf.Ticker('PFE')

pfe.info

...will allow access to all of this data:

{'52WeekChange': 0.025267482,

 'SandP52WeekChange': 0.16119564,

 'address1': '235 East 42nd Street',

 'algorithm': None,

 'annualHoldingsTurnover': None,

 'annualReportExpenseRatio': None,

 'ask': 36.55,

 'askSize': 4000,

 'averageDailyVolume10Day': 82095557,

 'averageVolume': 31018400,

 'averageVolume10days': 82095557,

 'beta': 0.641488,

 'beta3Year': None,

 'bid': 36.52,

 'bidSize': 1800,

 'bookValue': 11.744,

 'category': None,

 'circulatingSupply': None,

 'city': 'New York',

 'companyOfficers': [],

 'country': 'United States',

 'currency': 'USD',

 'dateShortInterest': 1604016000,

 'dayHigh': 37.42,

 'dayLow': 36.3,

 'dividendRate': 1.52,

 'dividendYield': 0.0422,

 'earningsQuarterlyGrowth': -0.714,

 'enterpriseToEbitda': 13.11,

 'enterpriseToRevenue': 5.206,

 'enterpriseValue': 253279617024,

 'exDividendDate': 1604534400,

 'exchange': 'NYQ',

 'exchangeTimezoneName': 'America/New_York',

 'exchangeTimezoneShortName': 'EST',

 'expireDate': None,

 'fiftyDayAverage': 35.149593,

 'fiftyTwoWeekHigh': 39.83871,

 'fiftyTwoWeekLow': 26.451612,

 'fiveYearAverageReturn': None,

 'fiveYearAvgDividendYield': 3.62,

 'floatShares': 5548891741,

 'forwardEps': 2.84,

 'forwardPE': 12.788733,

 'fromCurrency': None,

 'fullTimeEmployees': 88300,

 'fundFamily': None,

 'fundInceptionDate': None,

 'gmtOffSetMilliseconds': '-18000000',

 'heldPercentInsiders': 0.00046999997,

 'heldPercentInstitutions': 0.70032,

 'industry': 'Drug Manufacturers—General',

 'isEsgPopulated': False,

 'lastCapGain': None,

 'lastDividendDate': 1604534400,

 'lastDividendValue': 0.36053,

 'lastFiscalYearEnd': 1577750400,

 'lastMarket': None,

 'lastSplitDate': 1605571200,

 'lastSplitFactor': '1054:1000',

 'legalType': None,

 'logo_url': 'https://logo.clearbit.com/pfizer.com',

 'longBusinessSummary': 'Pfizer Inc. develops, manufactures, and sells healthcare products worldwide. It offers medicines and vaccines in various therapeutic areas, including cardiovascular metabolic and pain under the Eliquis, Chantix/Champix, and Premarin family brands; biologics, small molecules, immunotherapies, and biosimilars under the Ibrance, Sutent, Xtandi, Xalkori, Inlyta, Braftovi + Mektovi brands; and sterile injectable and anti-infective medicines under the Sulperazon, Medrol, Vfend, and Zithromax brands. The company also provides medicines and vaccines in various therapeutic areas, such as pneumococcal disease, meningococcal disease, and tick-borne encephalitis under the Prevnar 13/Prevenar 13 (pediatric/adult), FSME-IMMUN, Nimenrix, and Trumenba brands; biosimilars for chronic immune and inflammatory diseases under the Xeljanz, Enbrel, Inflectra, and Eucrisa brands; and amyloidosis, hemophilia, and endocrine diseases under the Vyndaqel/Vyndamax, BeneFIX, Genotropin, and Refacto AF/Xyntha brands. In addition, the company is involved in the contract manufacturing business. It serves wholesalers, retailers, hospitals, clinics, government agencies, pharmacies, and individual provider offices, as well as disease control and prevention centers. The company has collaboration and/or co-promotion agreements with Bristol-Myers Squibb Company and Astellas Pharma US, Inc.; a licensing agreement with Akcea Therapeutics, Inc; a strategic alliance with Verily Life Sciences LLC; collaboration agreements with Merck KGaA and Valneva SE; a clinical trial collaboration and supply agreement with IDEAYA Biosciences, Inc.; a material transfer and collaboration agreement with BioNTech SE; a clinical supply collaboration with Jiangsu Alphamab Biopharmaceuticals Co., Ltd; a research collaboration and license agreement with BioInvent International AB; and a multi-target drug discovery collaboration with Sosei Group Corporation. Pfizer Inc. was founded in 1849 and is headquartered in New York, New York.',

 'longName': 'Pfizer Inc.',

 'market': 'us_market',

 'marketCap': 201881092096,

 'maxAge': 1,

 'maxSupply': None,

 'messageBoardId': 'finmb_162270',

 'morningStarOverallRating': None,

 'morningStarRiskRating': None,

 'mostRecentQuarter': 1601164800,

 'navPrice': None,

 'netIncomeToCommon': 8686000128,

 'nextFiscalYearEnd': 1640908800,

 'open': 36.93,

 'openInterest': None,

 'payoutRatio': 0.96769994,

 'pegRatio': -12.73,

 'phone': '212 733 2323',

 'previousClose': 36.04,

 'priceHint': 2,

 'priceToBook': 3.092643,

 'priceToSalesTrailing12Months': 4.1497483,

 'profitMargins': 0.17854,

 'quoteType': 'EQUITY',

 'regularMarketDayHigh': 37.42,

 'regularMarketDayLow': 36.3,

 'regularMarketOpen': 36.93,

 'regularMarketPreviousClose': 36.04,

 'regularMarketPrice': 36.93,

 'regularMarketVolume': 62576363,

 'revenueQuarterlyGrowth': None,

 'sector': 'Healthcare',

 'sharesOutstanding': 5558400000,

 'sharesPercentSharesOut': 0.0083,

 'sharesShort': 45925049,

 'sharesShortPreviousMonthDate': 1601424000,

 'sharesShortPriorMonth': 45163995,

 'shortName': 'Pfizer, Inc.',

 'shortPercentOfFloat': 0.0083,

 'shortRatio': 1.93,

 'startDate': None,

 'state': 'NY',

 'strikePrice': None,

 'symbol': 'PFE',

 'threeYearAverageReturn': None,

 'toCurrency': None,

 'totalAssets': None,

 'tradeable': False,

 'trailingAnnualDividendRate': 1.5,

 'trailingAnnualDividendYield': 0.041620422,

 'trailingEps': 1.538,

 'trailingPE': 23.615084,

 'twoHundredDayAverage': 34.76005,

 'volume': 62576363,

 'volume24Hr': None,

 'volumeAllCurrencies': None,

 'website': 'http://www.pfizer.com',

 'yield': None,

 'ytdReturn': None,

 'zip': '10017'}

 

Tip -- Number conversions for non U.S. regional settings

---In smf_addin@yahoogroups.com, <prosuite2000i@...> wrote :

Best way to resolve regional settings (working for months) is ' Replace all occurences of CDec to RegionalCDec in you code. ' and keep CDec unchanged in RegionalCDec function ' Add in modUtilities Public Function RegionalCDec(ByVal varUS2Regional As Variant) As Variant Dim varRegionalNumber As Variant Dim strRegionalDecimalSep As String On Error GoTo RegionalCDec_Error If UCase$(varUS2Regional) = LCase$(varUS2Regional) Then     ' Likely a number         strRegionalDecimalSep = Mid$(1 / 2, 2, 1)     If strRegionalDecimalSep <> "." Then         ' Local is not US                 varRegionalNumber = varUS2Regional         If InStrB(varRegionalNumber, ",") <> 0 Then             ' Get rid of Thousand Separator             varRegionalNumber = Replace$(varRegionalNumber, ",", vbNullString)         End If                 If InStrB(varRegionalNumber, ".") <> 0 Then             ' Replace US Decimal Separator to Local             varRegionalNumber = Replace$(varRegionalNumber, ".", strRegionalDecimalSep)         End If                 'varRegionalNumber = CDec(varRegionalNumber)         If IsNumeric(varRegionalNumber) Then varUS2Regional = varRegionalNumber     End If End If RegionalCDec_Error: RegionalCDec = CDec(varUS2Regional) End Function Using CDate has the same side effects with Regional Settings. ' Replace CDate in RCHGetYahooHistory by RegionalCDate(vItem(0), "yyyy-mm-dd") ' Replace CDate in smfGetMSNHistory by RegionalCDate(vItem(0), "m/d/yyyy") ' Add in modUtilities Public Function RegionalCDate(ByRef varUS2Regional As Variant, Optional ByVal strDateFormat As String = "mm/dd/yyyy") As Date Dim varRegionalDate As Variant Dim lngLen As Long Dim strDay As String, strMonth As String, strYear As String Dim lngDay As Long, lngMonth As Long, lngYear As Long Dim lngPosDay As Long, lngPosMonth As Long, lngPosYear As Long Dim lngLenDay As Long, lngLenMonth As Long, lngLenYear As Long On Error GoTo RegionalCDate_Error lngLen = Len(varUS2Regional) If lngLen <> 0 Then     If UCase$(varUS2Regional) = LCase$(varUS2Regional) Then         ' Likely a date                 strDateFormat = LCase$(strDateFormat)                 lngLenDay = lngLen - Len(Replace(strDateFormat, "d", vbNullString))         If lngLenDay <> 0 Then             lngPosDay = InStr(strDateFormat, String$(lngLenDay, "d"))             If lngPosDay <> 0 Then                 strDay = Mid$(varUS2Regional, lngPosDay, 2)                 If Not IsNumeric(strDay) Then _                 strDay = Mid$(varUS2Regional, lngPosDay, lngLenDay)             End If         End If                 lngLenMonth = lngLen - Len(Replace(strDateFormat, "m", vbNullString))         If lngLenMonth <> 0 Then             lngPosMonth = InStr(strDateFormat, String$(lngLenMonth, "m"))             If lngPosMonth <> 0 Then                 strMonth = Mid$(varUS2Regional, lngPosMonth, 2)                 If Not IsNumeric(strDay) Then                     strMonth = Mid$(varUS2Regional, lngPosMonth, lngLenMonth)                     If Not IsNumeric(strMonth) Then strMonth = _                     (InStrB("JanFebMarAprMayJunJulAugSepOctNovDec", strMonth) + 5) \ 6                 End If             End If         End If                 lngLenYear = lngLen - Len(Replace(strDateFormat, "y", vbNullString))         If lngLenYear <> 0 Then             lngPosYear = InStr(strDateFormat, String$(lngLenYear, "y"))             If lngPosYear <> 0 Then strYear = Mid$(varUS2Regional, lngPosYear, lngLenYear)         End If                         varRegionalDate = DateSerial(strYear, strMonth, strDay)         If IsDate(varRegionalDate) Then varUS2Regional = varRegionalDate     End If End If RegionalCDate_Error: RegionalCDate = CDate(varUS2Regional) End Function --- On Mon, 10/11/10, Randy Harmelink <rharmelink@...> wrote: I think there are only two ways to overcome it: 1. Change to the US regional settings so the VBA function does the conversion correctly. 2. Change the add-in conversion to do the process a different way with another function. Since the method suggested to me for (2) didn't work for US regional settings, I didn't really look at it very long. See: http://finance.groups.yahoo.com/group/smf_addin/message/6537 On Mon, Oct 11, 2010 at 5:38 AM, po_zi_2000 <zetelsl@...> wrote: Do you know how to evercome this? Is there anything I can do to fix it in my own templates? Since I will normally work under European settings once fixed there won't be any conflicts with US settings.

Tip -- 9.5 - FAQ: What is the smfForceRecalculation macro and how do I use it?

---In smf_addin@yahoogroups.com, <rharmelink@...> wrote :

OverviewWhen the RCHGetElementNumber() and RCHGetTableCell() and RCHGetWebData() functions are invoked, they actually extract data from a copy of the web page that was saved by the add-in.  This is done to make the functions run more quickly, because the actual retrieval of a web page from the Internet is the part of the process that takes the most time.  For example, if you were to get 20 or 30 data items from the same web page, this process allows the web page to be retrieved once and then all extractions are done from that single retrieval of the web page from the Internet.  Otherwise, each invocation of those functions would need to grab a new web page.

In general, the process operates as follows:

• The "saved array" is first checked, to see if the web page the data is being extracted from has already been retrieved from the Internet.
• If an entry in the "saved array" is found, the data is extracted from the saved copy of the web page.
• If an entry is not found in the "saved array", the source code of the web page is retrieved from the Internet and then put into the "saved array", which is indexed by URL.  Then, the data is extracted from that saved copy of the web page.

What does the smfForceRecalculation macro do?

When you run this macro, it purges the "saved array" of web pages and then triggers a full recalculation within EXCEL.  Since the "saved array" of web pages has been emptied out, the add-in functions that use the "saved array" of web pages will no longer find a saved copy of a web page, and be forced to retrieve a fresh copy of the web page.  Note, however, that ONLY the first function invocation that uses a given URL will go to the Internet -- because it will save a new copy of the web page, which all other functions using that URL will use instead of going to the Internet to get the web page again.

In addition, this macro will also force all of the RCHGetElementNumber() element definitions to be reloaded.  That was actually the original reason the macro was written -- so I could modify an element definition and tell the add-in to reload the new element definitions and test them to see if the changes worked correctly.

How do I run the smfForceRecalculation macro?

You can use the keyboard shortcut alt-F8 to bring up EXCEL's macro dialog window.  Unfortunately, this window does NOT list macros that are part of an add-in.  That means you need to type in the macro name.  When you complete entering the macro name, you should see the various buttons activate.  You then click "Run" to run the macro.

I have actually assigned the macro to a keyboard shortcut (done from the "Options..." button, which is located several buttons below the "Run" button used above).  I also have added a button to my Quick Access Toolbar in EXCEL 2007 that runs the macro.  It is what I use most often.  In fact, I usually click on it each time before I open a new workbook, to reset the "saved array" of web pages for that workbook.  But you probably don't want to do that if you have a number of workbooks open that use add-in functions, since the macro will force ALL add-in functions in ALL workbooks to recalculate.

Additional Notes

The RCHGetYahooQuotes() and RCHGetYahooHistory() functions do NOT use the "saved array" of web pages.  That's because they both return an entire set of data with each invocation of the function.  So there is no advantage to saving the web page -- since all possible data is extracted from a single Internet access.  In fact, Yahoo is returning CSV files, not web pages.  So the functions just need to parse the CSV files and place the data into the range they were given access to when the formulas were array-entered.

Note, however, that the smfForceRecalculation macro WILL make those functions grab new copies of the CSV files and parse them out.

---

Tip -- 9.4.2 - FAQ: What do I need in addtion to smfForceRecalculation to get my data to refresh?

---In smf_addin@yahoogroups.com, <rharmelink@...> wrote :

Another possible reason data is not refreshing is that you have your Internet Options set up to cache web pages. Since EXCEL and the add-in use the same Internet access engine as IE, it also utilizes IE's cookies and "Temporary Internet Files" folder.

To check/change your page caching settings for the "Temporary Internet Files" folder:

1. Click on your system's "Start" button.
2. Click on the "Control Panel" option.
3. Double-click on "Internet Options".
4. In the "Browser settings" section, click on the "Settings" button.
5. Under where it says "Check for newer versions of stored pages:", you want to toggle the "Every time I visit the web page" radio button. Without this, the add-in would just retrieve IE's stored copy of the web page instead of the Internet access engine getting a fresh copy of the web page from the Internet.
6. Click "OK" to save the settings and exit
7. Click "OK" to exit
8. Close the "Control Panel" window.

Did you need to change your setting there?